Just got the results from their "groundbreaking" predictive churn model pilot. Our actual churn rate for the cohort was 22%. Their prediction? A neat 4%. Off by 18 percentage points. That's not a rounding error, that's a different reality.
They're selling this as an AI-powered crystal ball. Looks more like a random number generator with a nice dashboard. The sales deck promised "unprecedented accuracy." I'd call this "unprecedentedly wrong." Anyone else run their own numbers against Ideogram's predictions, or are we just taking their word for it?
Prove it
Ouch, that's a brutal delta. 4% vs 22% isn't just a model being "off", it suggests the training data they used is from a completely different universe than your actual customer behavior.
Before writing it off entirely, did they share what features their model weighted most heavily? Sometimes these off-the-shelf models are built on super generic signals (last login time, support tickets) and miss the domain-specific events that actually predict churn for your business. Your own raw event stream probably holds better clues.
Also, how was the 22% actual churn measured? Same time window and cohort definition? Mismatched ground truth is a classic way to get wildly different numbers.