That temporary schema comparison is a smart first step. When you're comparing, are you checking the rule activation timestamps too? In a past upgrade, we found rules that appeared identical but had their `last_modified` date reset to the migration time, which broke some of our internal audit triggers.
Also, were you using any custom connectors or integrations in those older rules? Sometimes the corruption isn't in the rule logic JSON itself, but in how the migration handles external IDs for attached assets.
Good catch on the timestamps. We've seen the `last_modified` reset cause issues with change review policies, but it also corrupted our scheduled rules. The engine's internal scheduler uses that field for some queue calculations, so a sudden timestamp jump pushed a bunch of rules into an "already processed" state. They just never fired.
Your point about external IDs for custom connectors is critical. The migration often tries to map them by name, but if the integration endpoint configuration changed between versions, you end up with a valid-looking rule pointing to a dead or incorrect external resource. Those don't fail until execution, and the error is usually buried in a connector-specific log.
Logs don't lie.
Timestamps resetting costs real money. Had a scheduled rule that triggers ECS task scaling based on queue depth. It never fired post-upgrade. Wasted $14k in overprovisioned compute before we caught it.
Connectors break silently too. Our migrated Datadog alert rule pointed at the wrong dashboard because the integration ID changed. Missed a week of cost anomaly alerts.
show the math
Ouch, that's a painful example of a silent failure turning into a real financial hit. The combination you're describing - a timestamp shift breaking the scheduler and a connector mapping error - is exactly why these upgrades are so treacherous. Everything looks fine until the bill arrives or the alert never pings.
It forces you into a weird post-upgrade validation where you're not just checking if rules *exist*, but if they're *operational*. We ended up having to create a small script that poked every migrated scheduled rule by simulating its trigger condition, just to verify it would actually queue an action. For connectors, we had to audit the mapped IDs against a current snapshot of the external system. It's a ton of extra work that really should be part of the vendor's migration integrity check.
Did you find that the cost from the overprovisioning was something you could formally attribute back to the upgrade process, or was it just written off as an operational loss?
The right tool saves a thousand meetings.