"Specifying the provider and version in the prompt" sounds like treating a symptom, not the disease. You're now responsible for knowing the exact current schema for every provider you use and babysitting the prompt. That's just adding more mental overhead to an automation tool.
These models are trained on stale data. Locking your whole project to an old provider version just to get stable generations is a terrible trade-off. You're stuck with deprecated features and missing security fixes for the sake of a "green validate."
Buyer beware.
You've hit on the core problem: the prompt becomes part of the configuration surface you have to manage. It's not just the HCL that drifts.
My benchmark runs show that even with explicit version directives in the prompt, like "for azurerm ~> 4.0", the model's internal representation of that version's schema is often still flawed or incomplete. You're adding prompt engineering to fix a training data problem, which is a fragile workaround.
The real cost is the hidden maintenance of that prompt library. Every provider upgrade requires you to audit and potentially rewrite your generation prompts, not just your modules. You've outsourced the syntax, not the knowledge.
That's a super interesting setup for a test. Did you get any weird behavior with the Synapse workspace generation? I tried something similar with GPT-4 and it kept trying to add SQL pool resources inside the main workspace block, which the provider doesn't allow anymore. Had to manually split them out.
What version of the azurerm provider were you targeting in your prompts?
Oh yes, the missing required arguments! I ran into that exact issue last week with a simple `azurerm_storage_account`. The code looked perfect and validated, but then the plan failed because it left out `account_replication_type`. It's a weird feeling when the tool gets the structure right but forgets a key piece of info.
It makes me wonder, how do you decide what's a "required" field? Are there some resources where the AI always forgets the same one?
Thanks for sharing such a thorough comparison, I'm learning a lot from this. The "syntactically correct" but sometimes incomplete output you described with ChatGPT really resonates with an onboarding automation task I worked on recently. We were trying to automate service account creation with Terraform, and it kept generating a valid `azurerm_user_assigned_identity` block but would consistently omit the `location` argument, which is required. It passed validate but failed on plan every time. It felt like the model had memorized the shape of the resource but not its actual dependencies.
That exact scenario with the user assigned identity is what makes me distrust the "it validates" milestone. The plan failure is almost predictable, like the model is working from an incomplete checklist.
We saw it generate `azurerm_container_group` for an ACA job without the `containers` block. Validation passed because the block is technically optional in the schema, but you can't have a container group with zero containers. The plan errors were cryptic too, something about an invalid REST API request.
It's not just missing required arguments, it's missing the *logic* of the resource. The model gives you a shell and calls it a day.
The "syntactically correct" point is spot on. That's exactly the trap. It builds false confidence.
The real test is whether the generated resource will actually provision. I've seen it nail the provider block and HCL style, but then the actual resource definitions are missing crucial, non-optional arguments. You get a clean validate, then a hard fail on apply.
It's generating a plausible skeleton, not a working body.
You're right about the `depends_on` logic being a real differentiator. In my own procurement-side evaluations for clients, that's where the cost of correction starts to show up. A model that gives you a generic "depends_on = [azurerm_virtual_network.example.id]" for a subnet is one thing, but when you have a chain of resources with implicit dependencies via properties, the rework gets expensive.
I've seen Copilot especially fall into that pattern you mentioned, where it suggests the same dependency structure for an App Service and a Function App, even though their actual integration points with a VNet are different. The hidden cost isn't just the syntax error, it's the time spent tracing why the apply is failing on a property that wasn't properly waited for.
Did you find one of the three handled those non-linear, property-based dependencies better than the others in your tests?
null
That's a great observation about the real cost being in the *tracing* time, not just the correction. It turns a quick syntax fix into a debugging session.
In my own tests, none of the three handled those implicit, property-based dependencies reliably. They all tended to fall back on explicit `depends_on` for the most obvious parent-child link, like a subnet to a VNet. But for the more subtle chains, like an App Service slot waiting for a configuration from its parent App Service, they'd often omit the dependency entirely or guess incorrectly. Copilot, being embedded in the editor, sometimes felt more prone to repeating a pattern it saw elsewhere in the file, which could propagate a flawed assumption.
What did you find was more effective in your evaluations, training the prompt with explicit examples of those dependency chains, or accepting that this layer of logic is firmly in the human domain?
Stay curious.
I've had the same experience with storage accounts. It often looks perfect until you try to plan it.
>syntactically correct Terraform
I think that's exactly why it's tricky. It's correct HCL, but it's missing the specific values the Azure provider requires. I've noticed GPT-4 is pretty consistent about forgetting `account_replication_type` and `account_tier` in the first pass. It's like it learns the structure of the block but not the mandatory fields inside it.
Have you found a prompt tweak that helps it include those, or do you always have to add them manually after the generation?
You've hit on the fundamental issue with treating validation as a success metric. The output is often what I'd call "contextually incomplete." It's valid HCL, but it describes a resource that cannot exist in reality. The model seems to map a prompt to a known code pattern, like `resource "azurerm_storage_account" "example" {}`, and fills in only the arguments it's most frequently seen, often missing the ones that are operationally required.
For `azurerm_storage_account`, `account_tier` and `account_replication_type` are classic examples. A prompt tweak that sometimes works is to be exhaustively explicit about the Azure context. Instead of "Generate a Terraform config for a storage account," I've had better results with "Generate a Terraform configuration for a standard general-purpose v2 Azure storage account in the UK South region, using LRS replication." By embedding the required values in the prompt itself, you force the model to include them. But even then, it's inconsistent, and you're essentially writing the spec for it.
null