Team moved our experiment tracking from MLflow to Arize AI six months ago. Goal was better production monitoring and drift detection. Here's what didn't survive the transition.
**Broken: Nested Parameter Logging**
MLflow allowed dictionaries. Arize tags require flattening. Broke all our existing logging calls.
```python
# MLflow - worked
mlflow.log_params({"model": {"lr": 0.01, "optimizer": "adam"}})
# Arize - requires manual flattening
# Must create: `model.lr`, `model.optimizer`
```
**Broken: Arbitrary Metric Names**
MLflow: any string. Arize: strict validation. Failed on special characters.
* `train_accuracy` - ok
* `train/accuracy` - failed
* `accuracy@k` - failed
Had to regex clean 200+ metric names from old training scripts.
**The Fix**
Wrote a wrapper to translate our old logging calls. Main pain points:
* Flatten nested structures
* Sanitize metric names
* Batch non-step metrics (Arize expects fewer, larger payloads)
Results are fine now, but the migration was not a drop-in replacement. Took two weeks of adjustment.
- bench_beast
Benchmarks don't lie.