Tried out that new AI-powered anomaly detection service. The idea is solid: it learns your baseline and pings you for "weird" stuff without manual thresholds.
The good:
* Setup took 10 minutes. Just pointed it at my Prometheus.
* It caught a subtle, slow memory leak I'd missed. Alert was useful.
The bad:
* It's a black box. Why did it flag that spike at 3 AM? "Our model detected an anomaly." Not helpful.
* False positive on a planned deployment. Marked it as "expected pattern" but the same pattern next week triggered again.
* Cost model is based on "analyzed metrics." They count every tag combination as a separate metric. Our `http_requests_total` by endpoint, method, status blew up the count.
Would I renew? No.
My rule: if I can't explain the alert to my team during an incident, I can't use it. This feels like outsourcing my brain, and when it's wrong, I have no way to debug.
For now, I'm back to simple SLO-based alerting with well-understood rules.
```yaml
# What I trust vs. what I tried
alert: ErrorRateHigh
expr: |
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m]) > 0.01
for: 5m
```
// chris
metrics not myths
Totally get the black box frustration. It's like having a super smart intern you can't ever ask "why?".
You mentioned the false positive on the planned deployment pattern. Did you try giving it any context, like a calendar of maintenance windows? I'm curious if feeding it more external data helps, or if that just moves the opacity problem somewhere else.
I'm still figuring out my own threshold for this stuff. I want the clever finds, but I need to at least follow the logic.
Ask me in a year