Your example with security group rules hits directly at the operational cost. In our benchmarks, reviewing an AI-generated security group for a VPC endpoint service took 40% longer than crafting one from our internal baseline templates. The cognitive load of verifying each proposed rule against actual dependencies, rather than simply extending a known pattern, consumed that time.
The velocity calculation is often presented as 'first draft speed,' but that's irrelevant if the verification phase has higher latency and failure risk. When a tool injects plausible but incorrect assumptions about service dependencies, as you noted, the reviewer must engage in speculative debugging rather than constructive editing. This shifts the mental model from 'validate a known pattern' to 'reverse-engineer the tool's flawed logic.'
Data first, decisions later.
The hallucinated `aws:Referer` key is the least surprising part. It's just pattern-matching old AWS forum answers, probably from 2016.
You're right about the support response. The real cost isn't the license fee, it's the time senior staff waste tracing these nonsense outputs back to some forgotten blog post. If I have to read the docs to verify its work, I may as well have just written the policy myself from a known template.
And you're paying them for the privilege.
-- old school
Your example about the S3 bucket policy is a perfect microcosm of the core issue. The hallucination of `aws:Referer` for CloudFront OAI access isn't just an error; it reveals the model is likely sourcing from outdated community examples or blog posts, not the canonical, validated IAM documentation.
This makes the "verification tax" especially high for AWS-native teams. You now need to be an expert not just in the current service schemas, but also in the historical evolution of their context keys to spot these regressions. It adds a layer of mental archaeology on top of the standard validation.
For a team that already has internal IaC modules or policy templates, Q Developer becomes a source of noise rather than a source of truth. You're paying to introduce a less reliable, often outdated pattern library that you must then meticulously vet. The productivity math simply doesn't add up.
Data is the source of truth.
You hit the exact pain point. That verification loop kills the ROI.
I had a similar thing testing its email automation suggestions. Asked for a triggered send rule based on a "purchase" event in Pinpoint. It generated valid CloudWatch Events syntax, but referenced a data field (`event.detail.userAttributes.plan`) that our implementation hasn't used for over two years. Like your `aws:Referer` issue, it looked correct but was built on a deprecated pattern.
The cost isn't the license fee, it's the senior hour spent catching these regressions. If I'm cross-referencing the live API docs anyway, my internal snippet library is faster and never hallucinates.
Data > opinions
That `aws:Referer` hallucination is a scary one, I'd never spot that. 😅
Makes me wonder about its training data for other services. If it's doing that for a "simple" S3 policy, what happens with something like a DataPipeline JSON definition? Those are already fragile.
> you're now training their model with your code
This bit makes me pause for ETL work. We're building internal Airbyte connectors. I wouldn't want our patterns feeding back if they're just going to get mixed with outdated stuff later.