Just built a replay system using Arize to test model upgrades

I’ve been working on a model upgrade pipeline for a real-time inference service (C++, some Python pre/post). The big risk, of course, is regression in performance or accuracy when swapping the model binary. I wanted a robust way to test the new version *before* pushing to production, so I built a replay system using Arize.

The core idea: capture live inference requests (with anonymized features) and store them as a dataset. Then, I replay that exact dataset against the *new* model version in a staging environment, logging everything to Arize. This lets me compare the two model versions side-by-side across all their telemetry.

Here’s the basic flow I scripted:

```python
# Simplified replay loop
for request in captured_requests:
# Old model (already has ground truth logged via Arize SDK in prod)
# old_prediction = old_model.predict(request)

# New model in staging
new_prediction = new_model.predict(request)

# Log to Arize with a distinct model_id and timestamp shift
client.log(
model_id="candidate-model-v2",
prediction_id=str(uuid.uuid4()),
features=request.features,
prediction=new_prediction,
actual=request.actual, # ground truth captured earlier
timestamp=request.timestamp + offset # avoid collision
)
```

I then used Arize’s dashboard to set up a comparison view:
* Performance metrics (accuracy, precision, recall) sliced by feature segments.
* Drift metrics (PSI) on predictions between the old and new model runs.
* A/B style charts for latency P99 (I added my own timing tags).

Some interesting findings:
* The new model had slightly better accuracy overall, but *significant* performance degradation on a specific category of users (~15% drop in recall). I would have missed that without the segmented analysis.
* Prediction distributions were stable (low PSI), so the issue wasn’t drift in the input—it was the model’s behavior change on that subset.
* The replay itself added overhead, but by batching the Arize log calls and using async, I kept it manageable.

Has anyone else set up something similar? I’m curious about:
* How you handled the ground truth capture for offline replay—did you store it separately or rely on Arize’s existing logs?
* Whether you tracked any system-level metrics (CPU/memory during inference) alongside the ML metrics, maybe using eBPF to correlate?
* If there are pitfalls with clock skew or ID collisions when replaying with timestamps.