Hot take: The default alerts are useless. You must customize them.

Just migrated our main model serving platform off their hosted offering and onto the self-managed Kubernetes setup. The whole process forced me to stare at their default alerting rules for a week. Verdict? They're borderline noise generators and will either miss real issues or spam you into alert fatigue.

The out-of-the-box Prometheus alerts you get from the Arize helm charts or terraform modules are built for the most generic, happy-path scenario. They assume your traffic is constant, your model outputs never drift, and your infrastructure is perfect. That's never the case. For example, the default "High Prediction Drift" alert might trigger at some arbitrary PSI threshold, but it doesn't consider whether the segment experiencing drift is business-critical or if it's a tiny population that's always noisy.

You **must** tear these apart and rebuild them for your actual SLIs. Here's a snippet of what we replaced the generic drift alert with. We needed to alert only on our core user segment and only during peak hours, and we combined it with a volume check to avoid false positives on low traffic.

```yaml
- alert: CoreSegmentDriftBusinessHours
expr: |
(az_metric_psi_drift{segment="core_user_cohort"} > 0.15)
and on(service_name)
(avg_over_time(az_inference_volume_total[1h]) > 100)
and (hour() >= 9 and hour() < 17)
for: 30m
annotations:
impact: "Drift on the primary user cohort exceeding threshold during business hours."
dashboard: "{{ $labels.dashboard }}"
```

The defaults also completely ignore orchestration failures. You get pod restarts, but no clear alert on repeated Argo workflow failures for batch runs, or on data ingestion pipeline latency from their collectors. We had to write our own based on their exported metrics.

* **What we scrapped or heavily modified:**
* Generic "High Data Drift" alert – now segmented and tied to business logic.
* "Low Feature Volume" alert – set dynamic thresholds based on day-of-week and time-of-day.
* Any alert that just checked if a metric was "non-zero" or "above X" without incorporating rate of change or context from other services.

* **What we had to build from scratch:**
* Alerts tying model performance drops (via Arize metrics) to specific Kubernetes node pool issues.
* Alerts for monitoring data pipeline backpressure (stuck messages in the collector).
* Alerts that combine Arize's `az_inference_latency_bucket` with our own service mesh latency metrics to pinpoint where degradation is happening.

The lesson here is to treat their alerting as a bare-minimum scaffold. The real work begins when you define what "broken" means for your business and instrument those conditions explicitly. Their defaults will not save you; they'll just give you a false sense of security.