I've hit all the same coverage gaps with that Atomic Red Team + playbook setup. The DAG approach others mentioned is the logical next step, but I found you can get 80% of the value by just wrapping your existing scripts in a simple Python orchestrator that tracks prerequisites.
My biggest headache was data pollution, so we solved it by routing all simulation traffic through a dedicated test VPC with unique tags. We use a separate Elastic index for those tagged events, which keeps the production dashboards clean and lets us nuke the whole dataset after a test run. The cost of the extra VPC is minimal if you tear it down after each test cycle.
For measuring latency, we added a step to our playbook that queries the test index for the alert by the simulation UUID and calculates the delta from the event timestamp. It logs the result to a CloudWatch metric, which gives us a latency trend over time without maintaining a separate spreadsheet. It's not perfect, but it surfaces rules with slow aggregation windows you'd otherwise miss.
terraform and chill
That's a smart approach to keep the data separate. We're doing something similar with a dedicated AWS account for tests instead of a VPC. Makes cleanup even easier with a full account teardown.
Your point about logging latency to CloudWatch is interesting. We've been storing the delta in a Dynamo table, but a metric would be way better for trending. How do you handle the alert query step if the rule fires but the alert itself is delayed by a slow dashboard? Do you just use the rule's internal trigger time from Elastic?