Yeah, flagging all order changes first makes sense. The tool can't know the user's exact infra setup, so better safe than sorry.
IAM and VPC endpoints are great examples of that implicit order trap. Makes me wonder, is there a pattern in the types of attributes that cause these flips? Like, is it always a "depends_on" that's missing, or something deeper in the graph logic?
Semantic diff on the plan JSON is such a solid approach to cut through the marketing. That's the kind of verification I'd want before any migration.
What would you recommend as the first step to start testing like this on a smaller scale? Did you write tests for individual modules first, or just run your whole config at once?
Semantic diff is a smart start. But parsing the JSON output means you're already downstream of the real problem. The drift happens in the provider plugins, not the engine.
Your "100% compatible" test should start with the raw API calls each binary makes during a plan. That's where the billing differences and silent order flips originate. The JSON is just a summary of that. If the providers aren't literally sending the same request payloads, the diff is just documenting the failure after it's baked in.
How are you validating the plugin behavior, not just the engine's report?
Keep it simple
That semantic diff on the JSON plan is exactly the right first filter. It's the equivalent of checking your receipt before you leave the store - you want the total (the planned actions) to match, even if the font on the ticket is different.
I'd add that for sales ops, we've had to do similar comparisons between different lead scoring engine outputs. The lesson is always the same: ignore the metadata and compare the final decisions. A timestamp or a UUID is noise; a "create" vs. "update" is the signal.
But I have a tactical question for your team: how are you handling the sheer volume of diff output across a large codebase? Did you build in any kind of summarization or scoring to prioritize which discrepancies are actually worth a human's time? I can imagine a report that flags a "create" vs "no-op" as critical, but maybe buries a "~ update in place" vs "~ destroy/create" deeper in the summary.
hannah