Hey everyone. We've been testing some MLOps tools to monitor our patient readmission risk model. Drift in healthcare data is no joke – shifts in demographics, seasonal trends, new procedures. We need something that catches issues fast but doesn't cry wolf.
Tried Arize's setup. The concept monitoring for key features like 'age' and 'length of stay' is straightforward. But I'm wondering if its statistical detection is sensitive enough for our compliance needs. Has anyone compared it to something like WhyLabs or Fiddler in a real clinical setting? Especially for PHI-safe pipelines.
What's actually working for your teams? Looking for specifics on alerting and root cause analysis. Cheers!
Always testing.
Hey, great question. That "cry wolf" problem is so real, especially when you're dealing with clinical compliance teams. They tune out fast if alerts are noisy.
We looked at Arize and Fiddler about a year back for our sepsis prediction models. Arize's statistical detection felt a bit generic for our use case too - like you said, it flagged shifts in 'age' easily but struggled to connect that to a meaningful performance drop on its own. Fiddler's explainability dashboard was stronger for root cause, honestly, but their PHI handling required a lot more upfront config on our end, which slowed us down.
What we actually ended up doing, and this might sound a bit janky, is using WhyLabs for the raw data drift monitoring on de-identified features because their logging is stupid simple, and then built custom business logic alerts on top. So if 'length of stay' drifts *and* our fallback model's performance dips by a certain threshold, that's a P1 page. Stitching tools together was the only way we got sensitivity without the false alarms.
Have you hit any specific compliance snags with Arize's data handling for PHI? That was a big sticking point for our legal team.
Automate all the things.