TIL: You can ask it to output in JSON format for easy parsing. Saves hours.

Great point about asking for JSON output. I've been using something similar for vendor risk assessments where I need structured data for our procurement checklist.

One tip: sometimes the AI still adds explanatory text before or after the JSON block, which breaks automated parsing. I've found adding "Only output valid JSON, no other text" at the end of the prompt helps get a cleaner result. Have you run into that issue with your AWS summaries?

It's made comparing vendor SLAs and pricing tiers much easier - I can pipe the JSON directly into a comparison spreadsheet template.

Yep, that text-around-JSON problem happens. The "only output valid JSON" trick is solid.

I've also had good luck specifying the schema directly, like including an example in the prompt. For instance: "Respond with JSON matching this exact format: {"metric": "string", "threshold": number}". It seems to lock in better.

Found it's still not 100% reliable with complex nested objects though. I usually wrap the call in a simple try/catch and retry once if parsing fails.

Relying on LLM JSON output directly in scripts sounds like a future post-mortem write-up. They're stochastic, not deterministic APIs.

For AWS summaries, you're better off just using the CLI with `--query` and `--output json`. It's actually reliable.

Example:
```
aws ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,InstanceType,State.Name]" --output json
```

It's structured, versioned, and won't suddenly give you a "Sorry, I can't do that" wrapped in curly braces.

Good point about using the CLI's native JSON. That's definitely the right call for existing resources.

But isn't the LLM trick more for planning or hypotheticals? Like asking for a cost estimate on a new architecture you're sketching out, where there isn't a CLI command yet. The AWS CLI can't tell you what something you haven't built might cost.

Still, I wouldn't pipe it directly into a script without a human check. The reliability issue is real.

The whole "planning and hypotheticals" angle is the worst possible time to use unchecked LLM output. If you're building a new architecture, you need reliable data for cost and security, not a hallucinated JSON object that looks plausible.

That's how you end up with a massive cloud bill or a compliance gap because the model confidently invented a cheaper, non-existent SKU. Use the pricing calculator API or actual vendor quotes.

Exactly. Using an LLM to generate cost data for architecture planning is asking for trouble. It's not just about wrong numbers - it's about creating false confidence. You'll get a neat JSON structure that looks authoritative, but the numbers inside are pulled from the model's training data, which is often outdated or generalized.

If you want to prototype, feed real pricing API data to the LLM and have it format that. Don't let it invent the data itself. The output is only as reliable as the input.

That's a really solid distinction you're drawing. Using the CLI for what exists and the LLM for what doesn't *feels* right. It captures the natural workflow.

I think the reliability concern you mention points to a useful middle ground. The prompt shouldn't be "estimate my cost," but something like "take this list of AWS services I intend to use, from this source, and format them into a JSON structure for our cost spreadsheet." That leans on the LLM for structuring, not inventing.

But you've put your finger on the core tension: the convenience of structured output versus the risk of trusting the content within that structure.

That's a good practical distinction. Using it for planning seems natural because you're in a gray area with no existing data to query.

But this makes me nervous for my own use case. I've been looking at CRM pricing and planning migrations. If I asked an LLM to output a JSON comparison of Salesforce vs Hubspot, how would I know it's pulling from current, accurate price sheets and not just old blog posts?

Maybe the risk is lower for qualitative pros/cons but the moment numbers get involved, I'd have to verify each line item anyway. So does the JSON formatting actually save time, or just make the output *look* more trustworthy than it is?

That's a clever use for scripting, getting structured output to avoid parsing headaches. I've seen similar approaches for generating draft configs or checklist items.

Have you noticed any variation in how well it follows your exact key names? Sometimes I need to be very explicit about data types too, like specifying if a value should be a string or number, to keep downstream tools happy.

It's definitely a time saver for turning freeform thoughts into something a script can handle.

You've nailed a key issue with that "draft config" use case. The key name and data type inconsistency is real. I've spent more time than I'd like fixing a script because an LLM output `"storage_gb": "500"` (a string) when my automation expected an integer, or it pluralized a key I defined as singular.

What's helped me is to include a single, concrete example right in the prompt. Not just a schema description. Something like: "Format the output as JSON. Example: {"service_tier": "Pro", "monthly_cost_usd": 299, "user_limit": 15}". That example sets the exact key names and, critically, shows the data types by example. It seems to anchor the output much better than abstract instructions.

It turns a neat trick into something almost reliable for that initial scaffolding, which is all I really want it for.

Totally agree you shouldn't trust it for actual cost numbers in a plan. The risk of a hallucinated SKU is too high.

But I've found it's pretty good for structuring the *framework* of a cost comparison when you're still brainstorming. Like, you ask it to "output a JSON template for a SaaS vendor comparison with fields for base price, user tiers, and contract term." Then you manually fill in the real numbers from the vendor's pricing page or a quote.

It gives you a solid starting structure without inventing the data.

You're right, it works. The key is using it for transformation, not generation.

I feed it `aws ec2 describe-instances` output and prompt "Convert this raw AWS CLI output into a JSON array with just these keys: instance_id, instance_type, state, private_ip". It structures existing data, doesn't invent.

Never let it generate the data itself, like a Terraform plan or cost estimate. That's how you get hallucinations in production. Use it to reformat CLI output you already trust.

You trust it to invent the schema, not the data. What if your "framework" includes a field for a license type or compliance feature that doesn't actually exist? You've baked a hallucinated requirement into your comparison.

Works for formatting, but your example shows the problem.

> "Output your answer as a JSON object with the following keys: summary, estimated_cost, security_considerations."

You're asking it to invent the estimated_cost. That's dangerous. The JSON just makes the hallucination look structured.

Use it to reshape *existing* data. Feed it your `terraform plan -json` output and ask it to filter. Don't let it generate the plan itself.