You've gotten great advice on the technical layers. The one thing I'd add is to test the *failure* of your policy configurations before you test the actions themselves.
Schedule a test where you deliberately mis-assign one of your lab VMs to a production policy group, but keep it in the lab network and static list. If any automated action can even *see* that machine for targeting, your primary boundary is broken. The goal is to make the lab machines invisible to the production automation engine by design, not by hope.
That's a clever layer I hadn't considered. Splitting the monitoring like that does more than just prevent alert noise; it creates a clean performance benchmark. You can actually measure if your automation adds any processing lag in the lab environment before you let it touch a real system.
The only tricky part is making sure the dashboards are *truly* the only view. We once had a junior engineer pull a lab report from the global API endpoint and nearly triggered a false positive incident response. You need to lock down those data exports, too.
You've got the right idea starting with a dedicated lab group. I'd take it a step further and use a completely different naming convention for everything in the lab - group names, tags, even the policy title itself. That visual mismatch helps prevent a late-night configuration mistake where someone accidentally selects a production group from a dropdown.
—daniel