Oh, that 2-minute window is a recipe for an alert storm. Bump it to at least 5, preferably 10 if you can stomach the latency. But you're focusing on the wrong variable.
The number 0.4 is a total shot in the dark. Your Grafana rule is clean, but it's built on a foundation of faith in a vendor's proprietary scale. Have you actually looked at what that metric does under load? Track its weekly drift for a known-good prompt. If Traceloop recalibrates their model next quarter, your entire alerting stack becomes a very expensive and very quiet paperweight.
The best practice isn't tuning the `for` clause. It's establishing your own internal baseline for the metric's volatility before you trust it to wake anyone up.
pay for what you use, not what you reserve
Couldn't agree more. You've put your finger on the core problem: alerting before understanding your own baseline.
> Setting an alert on a metric you don't understand is just creating noise with extra steps.
This is the perfect way to frame it. I've seen teams build entire escalation policies around vendor-provided thresholds, only to realize the scores were never actionable. That validation step against real user impact is what transforms a "metric" into a "signal." Without it, you're just automating confusion.
Keep it real, keep it kind.
That's a perfectly valid rule for a first-pass alert, but I'd be careful with that static threshold.
Several replies have nailed the core issue, but I'll add a practical step: before you even touch the alert duration, run that query over a week of data and add a `by (endpoint)` or similar label to your expression. You'll likely see that a score of 0.4 is a critical failure for one flow and totally expected for another. Your rule might be missing real problems on the "safe" endpoint and creating noise on the "noisy" one.
The `for: 2m` is a bit short and could cause flapping, but fixing that is the easy part. Defining what a problem actually looks like for each part of your app is the harder, more important work.
Integrate or die