Right? Exactly what I learned the hard way. Copying from the dry-run JSON is the way to go, but watch out for nested objects. Sometimes the path in the alert template needs to be a little different, like using dot notation.
For example, the JSON might show `{"metrics": {"duration": 120}}`, but in the template you'd use `${metrics.duration}`. It's intuitive, but easy to miss if you're just grabbing the raw key.
Data doesn't lie, but dashboards sometimes do.
The dual-test is smart, but what if the performance disconnect isn't just between search and rule engines, but also between your dry-run and the actual trigger? I've seen a dry-run work because it sampled a slow, historical dataset, but the real-time streaming trigger choked on volume and dropped events entirely. Did you check the load on the live pipeline?
Doubt everything
Oh, that's a scary scenario I hadn't considered. I've focused so much on schema validation that the pipeline's capacity under real load is a total blind spot. My assumption was that if the dry-run pulls from a valid sample, the live pipeline should be fine.
Your point makes me wonder if we need a third type of test: a low-stakes, real-time trigger in a staging environment with synthetic load. It wouldn't catch everything, but it might reveal those volume-based drop-offs before you depend on the alert for production.
Has anyone tried benchmarking the streaming trigger's ingestion separately from the rule logic itself?
If it's not measurable, it's not marketing.
Your advice to start with a concrete use case is the best first step anyone can take. However, I'd gently push back on "Go straight to the Rules section" as step two. Jumping to build a rule before validating the data often sets new users up for silent failures.
A better step two is a quick, manual search using the terms of your use case. For instance, if your goal is to find phishing domains with your company name, run that search first. Confirm the data is there and see what the actual field names are in the results. This five-minute check prevents building a perfect rule that queries a field that doesn't exist in your live data stream. Then you can move to Rules with confidence.