It's a decent formatting trick, but your `estimated_cost` example is a straight path to bad data. That's the model inventing numbers.
Use it to structure known outputs, not generate them. Feed it your actual `terraform show -json` or `aws ec2 describe-instances` dump first, then ask for JSON with your keys. Makes a clean summary without hallucinations.
For Terraform plans, I run `terraform plan -json | jq ...` directly. Safer than asking an LLM to interpret it.
Ship fast, review slower
Your approach with the estimated_cost key is dangerous for any operational decision-making. You're effectively asking a language model to act as a pricing API, which is pure hallucination territory. The cost for an instance type varies by region, purchase option, and even the hour of the day for Spot.
The utility is in formatting known data. The workflow should be: gather your actual state via CLI, feed that raw output as context, then instruct the model to reformat it into your specified JSON schema. That turns it into a structured view of your *actual* environment, not a plausible fiction.
null
That "Only output valid JSON" instruction is a lifesaver. But I've found the exact placement in the prompt changes the reliability.
Sometimes putting it *first* works better: "Respond *only* with valid JSON. Structure the output with these keys: ..." It frames the whole task differently. Depends on the model, though.
For vendor pricing, do you feed it the raw data sheet first? Or just ask for a comparison based on vendor names? I'm trying to gauge the hallucination risk.
Ask me about hidden egress costs.
That's a really good point about prompt structure. I've also found that putting the "Only output valid JSON" instruction first, and even repeating it at the end, can significantly improve adherence. It's like setting hard rails for the response.
On your vendor pricing question: feeding the raw data sheet first is absolutely mandatory. The risk otherwise is total fiction. I'll take a CSV from their website or a PDF spec sheet, paste it into the prompt with a clear delimiter, *then* ask for the JSON comparison. Even then, I validate a few key figures manually against the source. The model is great at reorganizing and presenting that data in a clean format, but I'd never let it recall or invent the numbers itself from just a vendor name.
For a while, I tried skipping the raw data step for simple "Feature X available in Product Y?" comparisons, but the models would confidently state features existed in plans where they didn't. It just sounds so plausible. Now, source data always comes first.
customer first