Okay, so I've been poking around Arize's documentation and testing their API for a project that monitors model performance. The term "drift" gets thrown around a lot, but I wanted to really understand *how* they arrive at that single percentage or score. It's not just a simple diff of averages.
From what I've gathered, Arize calculates drift primarily by comparing two distributions: a **baseline** (usually training or a known-good period) and a **production** window. The method depends on the data type:
**For numeric features (like a credit score):**
They typically use **Population Stability Index (PSI)** or **Jensen-Shannon Divergence**. Basically, they bin the data and compare how much the distribution of values has shifted between baseline and production. A high PSI = high drift.
```python
# Simplified conceptual example - they're doing something like this
from scipy.stats import entropy
import numpy as np
def js_divergence(p, q):
# Calculate Jensen-Shannon Divergence between two distributions
p = np.asarray(p)
q = np.asarray(q)
m = 0.5 * (p + q)
return 0.5 * (entropy(p, m) + entropy(q, m))
```
**For categorical features (like a loan type):**
They often use **Chi-Squared Test** or **Jensen-Shannon** again, comparing the frequency of each category. If 'Loan Type A' was 40% in training but is now 65% in production, that'll trigger drift.
The key things that tripped me up at first:
- You **choose the baseline**. It's not always the training set; sometimes it's last week's production.
- The **production window** matters. Is it the last hour? Last day? That drastically affects the calculation.
- They calculate it **per feature** and also for **model outputs/predictions**. So you can see *what* is drifting, not just *that* drift occurred.
Has anyone else dug into the specifics? I'm particularly curious about how they handle multivariate drift or if they offer different statistical tests via the API. The docs mention KL Divergence and Wasserstein Distance for some use cases too.
That's a helpful breakdown. I was also trying to figure out how they map the PSI or JSD result to that single percentage score you see in the dashboard. The documentation mentions thresholds, but I'm not clear on how those are set by default. Is it just a fixed rule, like PSI above 0.25 means 100% drift, or is it a more gradual scaling? And when you have multiple features, how is that rolled up into an overall model drift score? Is it an average, or does it weight by feature importance?
That's a solid starting point on the methods. One key detail is their binning strategy for numeric features, which heavily influences the PSI result. Arize uses quantile binning by default (not equal-width) to ensure each baseline bin has roughly the same number of observations. This prevents sparse bins from skewing the divergence metric.
For your categorical example, they primarily use Jensen-Shannon Divergence there as well. The crucial part is how they handle unseen categories in production, which they typically group into an "other" bucket to avoid infinite divergence.
Your conceptual code is close, but in practice they apply smoothing (like adding a small epsilon) to all bins to handle zeros before the entropy calculation.
The epsilon smoothing is a critical practical step, but its magnitude is nontrivial. A poorly chosen epsilon can artificially suppress real drift signals, especially in high-cardinality categorical features where many bins might have near-zero probability. I've seen teams set it as `1 / len(baseline_sample)` as a rule of thumb, but this can still be too large for massive baselines.
Also, while quantile binning prevents sparse bins in the baseline, it can create production bins with zero counts if the distribution shifts dramatically, which is exactly when you most need an accurate measure. The smoothing handles the math, but the choice of bin count becomes hyper-parameter sensitive in those edge cases.
Every dollar counts.