Exactly. Your test illustrates the real lever. People get stuck tweaking knobs on the tool when they should be curating the input library.
Your "night and day" result with Well-Architected reviews and logs is the blueprint. It goes beyond just adding cost data; you swapped marketing language for documents that are *structured around trade-offs and failure states* to begin with. That's the key shift in sourcing.
The tool isn't a critic. It's a reflector. Feed it a press release, you get fluff. Feed it a post-mortem, you get actionable insights.
Yeah that "reflector" analogy is spot on. It makes sense now why my earlier tests felt so hollow, I was just feeding it the AWS docs chapter on Lambda.
So the trick is building a library of failure states. Where do you even find good post-mortems? Are there public archives for that, or is it mostly internal stuff?
It's the source material, not the parameter. Whitepapers only describe the happy path.
You need to feed it the messy data. For Lambda vs Fargate cost insights, upload three things:
1. A screenshot of the Pricing Calculator for a spiky workload.
2. A blog post that actually measures cold start durations.
3. A real CloudWatch graph showing a traffic burst.
Then ask your same question. The shallow answer you got is just telling you your source library is all marketing fluff.
Your point about messy data is correct, but I'd stress that even blog posts and graphs need the right context. A screenshot of a Pricing Calculator is static data. It's a single point-in-time estimate. For a tool to analyze trade-offs, you need documents that contain *comparative* data.
For instance, you need the calculator outputs for the *same* workload modeled on Lambda, then on Fargate. Even better, include a third document showing the calculated cost after applying a 1-year Fargate Savings Plan. The tool can then surface the delta between the models because the comparison is explicitly laid out in the source material. It mirrors relationships you've already documented.
The CloudWatch graph is a good example. A graph alone shows a burst. But pair it with a developer's annotation explaining the resulting latency spike due to cold starts, and now you've given the tool the causal link it needs to reflect back. The source must contain the narrative of trade-offs.