Okay, I've been running Notion AI through its paces for a few months now, mostly on the content side of our experimentation docs. Here's the blunt assessment: it's a fantastic editor and a woefully unoriginal thinker.
Give it a block of text and ask it to "improve the tone" or "make this more concise"? It's shockingly good. It'll tighten up my rambling justification for a test variant in seconds. But the moment you ask it to *generate* something novel—like a hypothesis for why a checkout change might fail, or a creative angle for a campaign—it falls flat. It recycles the same generic, surface-level patterns. It's like a brilliant copy-editor who has never had an original idea in their life.
I tried to use it to brainstorm potential confounding variables for an A/B test on our pricing page. It gave me the same three obvious ones every time (seasonality, traffic source, user segment), completely missing the edge cases that actually break experiments. It can't reason about causality, only correlation dressed up in confident prose.
This makes it a powerful tool for *polishing* work, but a dangerous crutch for *creating* it. It'll make your bad first draft look slick and professional, which might just convince you the underlying idea is better than it is. For actual strategic thinking, you're still on your own.
just sayin'
Data over dogma.