Alright, so I finally got my Arize integration hooked up to our existing PagerDuty/Slack alerting pipeline last week. It was a bit more of a puzzle than I expected, mainly because Arize's own alerting is great for the platform, but I wanted everything centralized. Sharing what I learned in case you're on the same journey.
The key is using Arize's **webhook notifications** as your trigger point. You set up an alert within Arize (say, on a drift metric crossing a threshold), and instead of just using their UI, you configure it to hit a webhook. That webhook is your own small service or a serverless function (I used a Lambda). This function transforms Arize's payload into the exact format your existing pipeline expects.
Here’s the rough flow I built:
* Arize detects an anomaly and fires a webhook to my AWS Lambda endpoint.
* The Lambda parses the JSON, extracts the critical bits: model name, metric, value, threshold, and the direct link back to the Arize investigation panel.
* It then reformats this into a PagerDuty v2 event payload and triggers an incident.
* PagerDuty takes over—routes it, notifies the on-call, and the alert also mirrors to our #ml-alerts Slack channel via PagerDuty's Slack integration.
The main gotcha was mapping severity. Arize's alert severity doesn't 1:1 map to PagerDuty. I ended up setting all Arize alerts as "critical" in the initial webhook and letting PagerDuty de-duplicate and manage escalation based on our routing rules. Also, make sure your webhook service is highly available—you don't want to drop alerts because a Lambda is cold.
The result? Our MLOps alerts now live alongside our infra and app alerts. The on-call engineer gets a single pane of glass, and the Arize link gives them immediate context to start digging. It's a solid setup. Curious if anyone else has tackled this and if you found a simpler path, maybe using a tool like Zapier in the middle?
Ship fast, measure faster.
Using a Lambda for that translation step is clever. I'm curious, did you have to do much error handling for when Arize's webhook payload format updates? That's always my worry with these custom connectors.
Also, what format did you use for the Slack message? Just a simple text summary with the link, or something more structured? I'm trying to set up a similar flow for our team's monitoring.
Good point about the webhook payload format. I actually added a validation step in the Lambda that checks the incoming JSON against a schema before transformation. If it doesn't match, it fails safely to a dead-letter queue for review. It adds a bit of overhead but prevents silent failures.
For Slack, I went with structured blocks. It surfaces the model name, metric value, threshold, and time window in a compact format. The most useful part is making the Arize investigation link a clear button. That way the on-call engineer isn't hunting for it. Did you find a better structure for actionable alerts?
Your schema validation approach is sound. I've found that using a JSON Schema validator library with explicit versioning in the webhook URL path gives you both format safety and backward compatibility. For instance, `/webhook/arize/v1` versus `/webhook/arize/v2` lets you stage updates.
Regarding actionable Slack alerts, structured blocks are indeed the way to go. I've extended that concept by including a second, quieter button that links directly to our runbook for that specific model alert. This creates a two-click path: one to diagnose in Arize, another for the prescribed immediate mitigation steps if the issue is known. It reduces context-switching for the on-call engineer.
What validation library did you end up using? I've had mixed results with some of the Python offerings.