Another day, another vendor promising to revolutionize our analytics with their "AI-powered, end-to-end, single-source-of-truth platform." I swear, if I hear "leverage" one more time, I'll need a new keyboard.
The sales call script is always the same: slick demo, curated data, glowing case studies. They're selling a fantasy. Our job is to drag the conversation back to reality. The goal isn't to be difficult, it's to discover if their tool survives contact with the actual, messy world.
I've found the only way to do that is to ask questions that force specificity and reveal methodology. Skip the "what it does" and go straight to "how it works and for whom." For instance, when they claim "30% average lift in conversion," don't nod. Ask: "What was the median lift across your client base? Can you share the full distribution of results, including any negative or flat outcomes?" The average is often a happy story built on one or two outliers.
Dig into their benchmarking. "When you compare my site's performance to your 'industry benchmark,' what's the underlying cohort? Is it all your customers? Only those with similar traffic? How are you controlling for seasonality or different experiment velocities?" Most benchmarks are statistically worthless vanity metrics.
For any feature they highlight, ask for the failure mode. "Tell me about a time the attribution model fell apart. What data conditions break your journey mapping? How does the system handle a major tracking loss, like ITP changes?" If they say it's never happened, they're lying or their product is too new to trust.
Finally, get concrete about integration. "Show me your raw event table schema. What's the latency for a live API call versus a batch upload? What's the exact process for backfilling historical data if we switch logic?" The devil is in the data pipeline, not the dashboard.
The good sales engineers will appreciate these questions. The rest will flounder. Either way, you learn what you need to know.
Data skeptic, not a data cynic.