Hey everyone! 👋 Just had to share a workflow win that's been saving our team hours of manual digging every week.
We've been using Arize AI for monitoring our lead scoring and content recommendation models, and while the dashboards are great for spotting drift or performance drops, we kept hitting a wall: by the time someone checked Arize, a model issue might have already impacted a marketing campaign or lead routing for hours. The alerting is good, but we needed it to scream louder for critical failures.
So I finally connected Arize directly to our PagerDuty instance. Now, when Arize detects a significant spike in prediction errors or a sharp drop in our churn model's precision, it automatically triggers a PagerDuty incident. The setup was pretty straightforward using webhooks. The key was defining our thresholds carefully in Arize's monitors:
* **High Severity:** For our main lead scoring model, any precision drop >15% from baseline for two consecutive checks.
* **Medium Severity:** Feature drift (PSI) exceeding 0.25 on key fields like `company_size` or `industry`.
* **Low Severity:** Data quality alerts, like sudden spikes in missing values for new leads.
The magic is in the correlation. The PagerDuty incident includes a direct link to the exact Arize dashboard and the specific slice of data that's problematic. Our on-call data scientist doesn't have to hunt—they can immediately start diagnosing.
Has anyone else set up similar integrations? I'm curious:
* What thresholds or metrics do you find most actionable to alert on?
* Any pitfalls to watch for with alert fatigue? We're still tuning our sensitivity.
It's made our model ops feel so much more proactive. No more "oops, that email segment went out with bad recommendations yesterday" moments!
Keep it simple.
Webhooks are a start, but you're still relying on Arize's polling interval. If it checks hourly, you've still got an hour of bad leads. Have you considered streaming your inference logs directly into your own monitoring stack? You could calculate precision on a sliding five minute window with something as simple as Flink or a Postgres hypertable, and then have sub-60-second alerting without the third-party lag.
Also, be careful with those PSI thresholds on fields like `industry`. A big new enterprise deal can legitimately skew that distribution overnight and trigger a false alarm. You might want to segment your monitors by lead source or campaign to avoid waking someone up for good news.
That's a great idea. How did you decide on that 15% precision drop threshold? We use lead scoring too, and I worry about seasonal dips looking like model failure.