Migrated from Arize AI to a custom Grafana solution - what we saved and lost

After a 14-month production deployment of Arize AI for monitoring our machine learning inference pipelines, our team recently completed a migration to a fully custom observability stack centered on Grafana, Prometheus, and Loki. The decision was driven primarily by long-term cost projections and a desire for deeper integration with our existing infrastructure. While the outcome has been largely positive for our specific use-case, the trade-offs were substantial and worth documenting for any engineering team considering a similar path.

Our primary metrics stack now consists of:
* **Prometheus** for scraping numerical metrics from our inference services and data pipelines.
* **Grafana** for visualization, dashboards, and alerting.
* **Loki** for aggregating and querying inference logs (e.g., model versions, input hashes, latency traces).
* A custom **Python service** that calculates drift metrics (PSI, KL Divergence) on scheduled windows and exposes them as Prometheus gauges.

The immediate and most tangible gain is financial. Arize's pricing model, based on "observation" volume, became a significant operational cost as our inference throughput scaled. Our annual commitment was projected to exceed $85k. The fully loaded cost of our new stack, including the incremental cloud compute for the monitoring services and the engineering hours amortized over three years, is approximately $22k per year. This represents a ~74% reduction.

However, the engineering investment was non-trivial. Replicating Arize's core functionality required approximately six person-weeks of senior data engineering time. The main development efforts were:

1. **Drift & Performance Calculation Engine:** Arize's automated statistical calculations and comparisons against a defined baseline had to be rebuilt.
```python
# Simplified snippet of our PSI calculation service
def calculate_psi(current_distribution, baseline_distribution, feature_name):
# Bin distributions (ensuring identical bins)
bins = np.histogram_bin_edges(baseline_distribution, bins='auto')
base_perc = np.histogram(baseline_distribution, bins=bins)[0] / len(baseline_distribution)
current_perc = np.histogram(current_distribution, bins=bins)[0] / len(current_distribution)

# Add epsilon to avoid log(0)
psi_value = np.sum((current_perc - base_perc) * np.log((current_perc + 1e-9) / (base_perc + 1e-9)))
return psi_value
```
2. **Context-Rich Dashboards:** Building Grafana dashboards that combined latency (Prometheus), business metrics (PostgreSQL query), and drift alerts into a single pane of glass.
3. **Data Pipeline Integration:** Modifying our inference logging to emit structured logs to Loki and Prometheus metrics in a consistent schema.

What we lost, or more accurately, what we now maintain ourselves, is the seamless "batteries-included" experience. Specifically:
* **Automated Root Cause Analysis:** Arize's correlation engine between model performance degradation, feature drift, and data pipeline issues is now a manual process of cross-referencing multiple dashboards and logs.
* **Out-of-the-Box Workflows:** The pre-built workflows for champion/challenger model comparison or embedding drift visualization do not exist. We must design and build these analyses ad-hoc.
* **Vendor Support:** When a metric behaved unexpectedly, we could lean on Arize support. Now, any anomaly in our drift calculations triggers an internal investigation into our own code.

In summary, the migration was unequivocally the right financial decision for our scale and engineering capacity. It provided ultimate flexibility and cost control. However, it exchanged a managed, product-centric experience for a toolkit that requires continuous maintenance and deep in-house expertise. Teams without sufficient data engineering and ML ops bandwidth should carefully weigh the hidden costs of building versus buying.

-- elliot