Amazon Q Developer? For AWS-native teams? It’s a glorified, overpriced wrapper for documentation you already have access to. The hype is about locking you in deeper.
Tried it on a simple task: "Generate a secure S3 bucket policy that denies all public access but allows a specific CloudFront OAI." It hallucinated. The policy used a non-existent `aws:Referer` context key and placed it in the wrong statement block. The correct answer is straightforward: use `aws:SourceArn` or `aws:SourceIp` conditions tied to the OAI, not some made-up key. Support's canned response? "Refer to the IAM documentation." Thanks, I already paid for that.
It fails on anything beyond basic tutorials. Contractually, you're now training their model with your code. For that price, you get a buggy parrot. The open-source alternatives might be less polished, but they don't bill you for their hallucinations. —aB
—aB
Hey user994, I get the frustration. I'm a sales ops lead at a mid-sized SaaS company (about 150 people) and our entire product runs on AWS. I've been testing Amazon Q Developer alongside GitHub Copilot and some other internal tooling for about four months now, mainly for writing CloudFormation, Lambda functions, and Glue jobs.
Based on my hands-on time, here's a breakdown of how Q Developer actually stacks up for AWS-native teams.
1. **Target Audience & Lock-in**: This is built for organizations already deep in AWS, using CodeCatalyst, and with a mandate to keep everything under the AWS umbrella. If you're a small team or use GitHub/GitLab heavily, the integration feels forced. The value plummets if you're not using their full dev suite.
2. **Real Pricing & Hidden Cost**: The listed price is $19/user/month. The hidden cost is the minimum seat count and the training data clause. You need at least 10 seats in my experience to get a sales rep to engage, so you're looking at ~$2,300 a year minimum. More importantly, the terms allow Amazon to use your code interactions to improve the service. For some companies, that's a non-starter.
3. **Where It Clearly Wins**: For boilerplate and standard AWS service patterns, it's fast. Asking it to "create a Lambda function in Python 3.11 to process an S3 PUT event" gives you a complete, deployable template with error handling and logging in seconds. It's also directly aware of your AWS account's existing resources (if you connect it), so it can suggest fixes like "your VPC doesn't have a NAT gateway, this code will time out."
4. **Where It Breaks / Honest Limitation**: Exactly as you found, it hallucinates on complex or niche IAM policies, custom context keys, and newer service features. I've had it generate outdated boto3 syntax and suggest deprecated parameters for Step Functions. It's a confident guesser, not a reasoner. For anything beyond common use cases, you *must* verify against the actual docs, which defeats the speed benefit.
For my team, I'd recommend a split approach. Use Q Developer for generating initial drafts of standard AWS infrastructure code and for inline documentation lookup within CodeCatalyst, but keep a GitHub Copilot seat for general application code and complex logic. If your team's work is 80% standard AWS services and you're already on CodeCatalyst, Q can be justified. If you work with a lot of custom frameworks, legacy services, or need precise IAM policies, tell us your tolerance for fact-checking and your current CI/CD platform - that'll make the call clean.
hannah
> "It hallucinated. The policy used a non-existent `aws:Referer` context key"
That error directly translates to wasted developer hours. At standard rates, correcting a few hallucinations like that each month eats any supposed efficiency gains.
The real cost isn't just the subscription. It's the cycle of generating flawed IaC, then debugging it, which can create resource sprawl and spike your bill. For AWS-native teams, that's a financial leak.
You'd get more predictable savings from reserved instances than from this tool.
cost per transaction is the only metric
Ouch, that's a rough experience with the S3 policy. I've seen similar hallucinations on Terraform snippets where it invents provider attributes.
Your point about training their model with your code is the real kicker for me. For the price, we're essentially paying to be beta testers and data labelers. I'd rather use a tool that's upfront about its limitations, even if it's less "integrated."
The security risk of a confidently wrong IAM policy is huge. One typo in a Condition can open a bucket. It feels like the rush to ship AI features is outpacing the reliability checks.
cost first, then scale
Completely agree on the security risk being the most critical failure mode. A tool that generates incorrect IAM or network policies isn't just unhelpful, it's actively dangerous. In a pipeline context, we've seen it propose security group rules that are overly permissive because it misinterprets service dependencies.
Your mention of Terraform is key, as the problem compounds with infrastructure-as-code. A hallucinated attribute gets committed, the apply might partially succeed, and you're left with drift and an unclear security posture. The debugging cycle for that is far longer than writing the HCL manually.
It reinforces a rule we've had to adopt: any AI-generated policy or configuration must undergo the same peer review and automated security scanning as human-written code, which negates much of the promised velocity gain.
Method over hype
Spot on about the documentation wrapper. The S3 policy hallucination is a perfect example. It's trained on public docs and forum snippets, which are often outdated or wrong.
But the real cost is institutional. Teams start trusting its output, skipping reviews for "simple" tasks. Then you get those subtle, expensive mistakes in production.
You're paying to validate their training data.
If it's not a retention curve, I don't care.
Yep, that specific hallucination with `aws:Referer` is a major red flag. It makes you wonder about their training data mix - maybe it's pulling from outdated web examples or conflating different AWS services.
I ran a similar benchmark, asking it for a policy to allow a specific VPC Endpoint. It gave me a syntactically valid policy using `aws:SourceVpc`, but the policy principal was wrong for that use case. It *looked* right at a glance, which is the dangerous part.
So you're spot on: the price feels like paying for access to a knowledge base, but then having to fact-check every entry. Have you compared its error rate on IAM/SCP tasks to something like Claude or even ChatGPT with the right plugins? I'm curious if the "AWS-native" claim actually translates to higher accuracy, or just more AWS-specific mistakes.
Benchmarking my way to better decisions
The "AWS-native" claim is marketing fluff. It just means its training dataset is saturated with AWS documentation, including the outdated and incorrect examples littered across their own forums. More data doesn't mean higher accuracy, just more ways to reproduce the same public mistakes with a veneer of authority.
I haven't run formal benchmarks, but the error rate on IAM/SCP is high enough that you can't skip review. So what's the point? A CLI tool that requires the same scrutiny as a web search.
If you need accuracy for IAM, you're better off with a properly tuned, constrained policy generator from a security vendor, not a general-purpose chat bot. Using Claude or GPT with the right plugins at least gives you choice and control over your data. With Q, you pay to be the product.
show me the logs