"Specifying the provider and version in the prompt" sounds like treating a symptom, not the disease. You're now responsible for knowing the exact current schema for every provider you use and babysitting the prompt. That's just adding more mental overhead to an automation tool.
These models are trained on stale data. Locking your whole project to an old provider version just to get stable generations is a terrible trade-off. You're stuck with deprecated features and missing security fixes for the sake of a "green validate."
Buyer beware.
You've hit on the core problem: the prompt becomes part of the configuration surface you have to manage. It's not just the HCL that drifts.
My benchmark runs show that even with explicit version directives in the prompt, like "for azurerm ~> 4.0", the model's internal representation of that version's schema is often still flawed or incomplete. You're adding prompt engineering to fix a training data problem, which is a fragile workaround.
The real cost is the hidden maintenance of that prompt library. Every provider upgrade requires you to audit and potentially rewrite your generation prompts, not just your modules. You've outsourced the syntax, not the knowledge.
That's a super interesting setup for a test. Did you get any weird behavior with the Synapse workspace generation? I tried something similar with GPT-4 and it kept trying to add SQL pool resources inside the main workspace block, which the provider doesn't allow anymore. Had to manually split them out.
What version of the azurerm provider were you targeting in your prompts?
Oh yes, the missing required arguments! I ran into that exact issue last week with a simple `azurerm_storage_account`. The code looked perfect and validated, but then the plan failed because it left out `account_replication_type`. It's a weird feeling when the tool gets the structure right but forgets a key piece of info.
It makes me wonder, how do you decide what's a "required" field? Are there some resources where the AI always forgets the same one?