Alright, who else is getting absolutely spammed by Arize alerts since their UI "upgrade"? My Slack channel looks like a warzone. It's not that our models suddenly got worse overnight—it's the monitors.
Seems like they've tweked the default thresholds or aggregation logic behind the scenes. My `drift_psi` monitor for a perfectly stable BERT model is now screaming bloody murder every hour.
```yaml
# Example of my now-useless monitor config (anonymized)
metric: drift_psi
condition: > 0.15
time_window: 1h
aggregation: mean
```
Nothing changed in my code or data pipeline. The only variable is Arize pushing a UI refresh. Feels like they're now calculating over a different window or the bucket logic is off.
Anyone dug into the API responses to see if the actual metric values changed, or is this purely a threshold/alerting bug on their end? Need to know if I'm rebuilding all my monitors from scratch or just waiting for a hotfix.
Yes, we've seen the same spike across three of our production monitors. The API response for the metric values hasn't changed, at least for our `accuracy` and `latency` checks. The numbers are consistent with our internal logs.
The issue appears to be in the alerting subsystem's evaluation logic. A monitor set with `condition: > 0.15` on a stable metric reading 0.12 is now triggering, which suggests a bug in how the condition is parsed post-update. It's not a threshold change, but a break in the comparator.
I'd recommend checking your alert history against the raw metric values from the API for a single firing. If they don't align, it's a platform bug and you shouldn't rebuild anything. Our vendor manager has already opened a severity-1 ticket citing a breach of service reliability.
Check the SLA.
We've isolated it to the `aggregation: mean` logic specifically. Our monitors with `aggregation: max` are fine, but the hourly mean calculation is now incorrectly weighting the data points within the window, likely due to a change in how partial buckets at window boundaries are handled. For a 1h window with 5-minute data points, you'd expect 12 samples, but the API is returning a mean derived from a different count.
You can verify this by comparing the monitor's triggered value against a manual mean calculation from the raw metric timestamps in the API for the same window. In our case, the manual calc was 0.11, but the alert claimed 0.16. It's a backend computation bug, not a threshold change. Don't rebuild, but you might need to switch to `max` as a temporary workaround for stability.
data is the product
Switching to `max` as a workaround is a terrible band-aid. It might stop the noise, but it fundamentally changes the alert's meaning. You're now alerting on transient spikes you might not care about, while potentially missing sustained low-grade drift.
Your partial buckets hypothesis is solid. I ran the same check on our end. The bug is even weirder: for our `drift_ksi` monitor on a 1h window with 1m data points, the API returned a correct list of 60 values, but the monitor's evaluation used a *subset* of them. The manual mean from the full list matched our logs; Arize's computed mean was 30% higher. It's not just weighting, it's sampling.
You can't just trust the monitor's claimed value. You have to pull the raw metrics for the exact firing window and compute it yourself. Until they fix it, we're scripting our own alert checks against the API and muting their monitor alerts.
-- bb
Good catch on the sampling issue - that moves it from a simple aggregation bug to a data pipeline problem. The partial bucket theory made sense, but an inconsistent subset of points is more serious.
Your scripting workaround is pragmatic, but it's creating alert logic debt. You're now maintaining two systems for the same check. That's the real risk with this bug - it's forcing teams into fragile, temporary solutions that become permanent. Has your vendor given a timeline for a fix?
Keep it civil, keep it real.
Your guess is right, the API metric values are unchanged. The bug is in the alert engine's aggregation. Don't rebuild your monitors.
Your `aggregation: mean` config is the trigger. The engine is incorrectly sampling data points within the window. Check your alert's claimed value against a manual mean from the raw metrics for that exact firing window. You'll see a mismatch.
slow pipelines make me cranky
Right, the API numbers being stable means the bug is isolated to their internal alert engine's compute step. That's the only good news here.
If you're already pulling raw metrics to manually verify, you might as well script a temporary validation layer. Call the metrics API for the last hour, compute the mean yourself, and compare it to the monitor's triggered value. Log the delta. It's extra overhead, but it gives you an audit trail for when you ask for SLA credits.
Switching to `max` is a last resort. It changes the semantics of the alert completely.
Integration is not a project, it's a lifestyle.
Exactly right about the mismatch. We did that manual check and the delta was huge.
Here's a quick Python snippet we're using to document the bug while we wait for the fix. It pulls the raw metrics for the last window and compares to the alert value from the webhook payload.
```python
# Quick sanity check for Arize's broken mean
import requests
import json
# Get your API key and space ID
ARIZE_API_KEY = "your_key"
SPACE_ID = "your_space"
METRIC = "drift_psi"
MODEL_ID = "your_model"
# Assume you have the alert's timestamp and claimed value
alert_time = "2023-10-05T14:00:00Z"
claimed_mean = 0.16
# Fetch raw metrics for the hour leading up to the alert
response = requests.get(
f"https://api.arize.com/v1/metrics?model_id={MODEL_ID}&metric={METRIC}&start={...}&end={...}",
headers={"Authorization": f"Bearer {ARIZE_API_KEY}"}
)
data_points = response.json()['data']
calculated_mean = sum([dp['value'] for dp in data_points]) / len(data_points)
print(f"Arize claimed: {claimed_mean}")
print(f"Actual mean: {calculated_mean}")
print(f"Delta: {claimed_mean - calculated_mean}")
```
Sending these diffs to support has been helpful. It moves the conversation from "our monitors are noisy" to "here's your bug, reproduced."
Keep deploying!
That script is a smart move for building a case with support. We've been doing something similar but sending the diffs to a dashboard instead, just to track the error rate over time.
One thing we noticed is that the size of the delta seems to correlate with the number of data points in the window. For our 1-hour windows with 5-minute data, the bug is small. For 1-hour windows with 1-minute data, it's enormous, which aligns with the 'sampling a subset' theory. Have you seen any pattern like that in your diffs?
api first
The API metric values are unchanged. It's an aggregation bug in their alert engine.
Don't rebuild your monitors. Check the alert's claimed value against a manual mean from the raw metrics for the exact firing window. You'll see a mismatch, likely due to incorrect sampling within the time window.
Opening a severity ticket with that evidence is the correct next step.
Trust, but verify
Agreed on the audit trail for SLA credits. We're logging those deltas directly to a Slack channel alongside the monitor ID. Makes it impossible for support to ignore.
One caveat: scripting that check adds a new failure point. If their API has a hiccup during your verification call, you could miss documenting a real bug. Still worth it though.
—b
Oh wow, I'm seeing this too and was just about to post! My Slack is a mess and I thought it was something I did wrong. We just set up Arize last month.
> Anyone dug into the API responses to see if the actual metric values changed
Thanks for asking this - I was scared to check because I'm still learning the API. I just did a quick manual check for one alert and the raw numbers look normal, but the alert value was way higher. So it seems like it's not our data.
Should I be opening a support ticket for this, or is everyone just waiting? Feels weird for a new user to be the one reporting a major bug.
>creating alert logic debt
Exactly. We're doing the same verification script, but it's not just a temporary patch. It's now a permanent fixture in our runbook because the vendor's fix timeline is "we'll update you when we have more information." Classic.
Our script now automatically opens low-priority tickets in our Jira support project with each delta over 5%. It's creating more process to compensate for a broken process. The irony isn't lost on me.
Same exact issue here with our drift monitors. The API values are fine, so you don't need to rebuild.
But don't just wait. Log the delta between the alert's claimed value and a manual mean from the API for that window. We're doing that and tagging it onto every support ticket.