Am I configuring this wrong, or is the insight generation just shallow?

You're not using it wrong. The `detail_level` parameter is a verbosity control, like others have said, not a depth control. It can't invent the cost implications you're after if they're not in your source.

I run into this in procurement all the time. If I only feed a vendor's own RFP response into a summary tool, I get a polished repeat of their sales points. To get the nuanced trade-offs, I have to curate the input: the vendor's doc plus our internal security review notes, the pricing annex, and a third-party comparison blog.

For your case, try feeding the whitepaper alongside the AWS Pricing Calculator page for Lambda/Fargate and a known benchmark article on cold starts. You're manually building the "balanced corpus" the tool needs to collate from. It's extra setup, but it's the only way to get past the shallow, promotional layer.

That compliance example is exactly the kind of nuance I was missing. It's not just about pulling together different documents, it's about the specific gap between a rule and its implementation in your environment.

When you mention mapping `aws_s3_bucket.logging` to the CIS control manually, it makes me wonder: could you trick the tool into doing that analysis by also feeding it a mapping document? Like a separate file that explicitly links Terraform resource attributes to control IDs? Or would it still just retrieve and rephrase those links without understanding the *absence* of the attribute in your actual code?

You've correctly identified the configuration, but you're expecting analytical synthesis from a retrieval tool. The `detail_level="high"` parameter only increases the word count of the retrieved text chunks; it doesn't enable critical thinking.

Your example with Terraform provider docs is telling. If you ask it to compare `aws_lambda_function` to `aws_ecs_task_definition` based only on the HashiCorp docs, you'll get a feature list, not an architectural trade-off. The tool lacks a model of "cost" or "performance" unless you provide documents where those concepts are explicitly discussed in relation to those resources.

For your AWS scaling question, you'd need to build a corpus that includes the opposing viewpoints: the whitepaper, the pricing pages, and third-party benchmarks. The tool can then retrieve and concatenate those points, but it won't infer the implications of spiky traffic on its own. That synthesis is still your job.

Exactly, and that's the crux of the vendor evaluation problem. Even if you build this ideal "corpus" of opposing viewpoints, you're assuming you already know which documents contain the critical counterpoints.

> The tool can then retrieve and concatenate those points

But who curates that list? The procurement team that already knows about the notorious cold start benchmark? This creates a blind spot. It reinforces known unknowns and misses the "unknown unknowns" a critic would spot. You've automated a search for what you already know to search for.

It's a polished echo chamber.

You've correctly identified the configuration, but you're expecting analytical synthesis from a retrieval tool. The `detail_level="high"` parameter only increases the word count of the retrieved text chunks; it doesn't enable critical thinking.

Your example with Terraform provider docs is telling. If you ask it to compare `aws_lambda_function` to `aws_ecs_task_definition` based only on the HashiCorp docs, you'll get a feature list, not an architectural trade-off. The tool lacks a model of "cost" or "performance" unless you provide documents where those concepts are explicitly discussed in relation to those resources.

For your AWS scaling question, you'd need to build a corpus that includes the opposing viewpoints: the whitepaper, the pricing pages, and third-party benchmarks. The tool can then retrieve and concatenate those points, but the synthesis is just adjacency. It can't infer the cost implication of a spike unless a document explicitly states "spiky traffic costs more on Fargate due to minimum task count."

Right, that detail_level parameter got me too when I first tried it. It's just about how much of the surface text it repeats, not depth.

You're hitting the same wall I did with CDN configs. If you only feed the vendor's own best practices doc, you'll get a summary of their marketing. For the cost trade-offs you want, you have to feed it the counterpoints manually - like that Cloudflare vs Fastly pricing blog from last year, or your own billing data.

It's a fancy search engine, not a critic. You have to build the debate yourself first.

Oh, that's a really interesting test you did! The part about the outputs being structurally identical even with different retrieval methods is kind of eye opening. It really shows the limit is in the design, not just the data you feed it.

So even if you manually add the pricing page and a benchmark, it's still just stitching sentences together? It can't actually connect the dots to answer a real question like "can my budget handle this?"

That makes me think of trying to use a report tool in Salesforce. If you only pull in the "Opportunity" object, you'll just get a list of deals. To see if you'll hit quota, you have to be the one to build the relationship to "Forecasts" and "Quotas" and ask the right question. The tool just shows the data you already linked together.

I guess the AI is doing the same thing, just with text chunks instead of database objects.

You've perfectly described the financial calculus these tools miss. The Lambda pricing page and your CloudWatch logs are just data points. The insight is in the delta between them, which requires an actual cost model.

I ran into this with DynamoDB on-demand versus provisioned capacity. The tool could regurgitate the pricing for a write request unit, but it couldn't take a week of application metrics and plot the crossover point where provisioned capacity becomes 40% cheaper. You have to build that spreadsheet yourself, mapping the irregular burst pattern to the monthly commit.

It's the difference between a dictionary and a translator. The tool has all the words, but it can't construct the sentence "This pattern is expensive."