Totally get that feeling. It's like the high score on the chart sets an expectation that doesn't match the real integration work. Even in my basic docker projects, the "quick start" guide never mentions the hours you'll spend debugging a port conflict or a volume mount.
Your point about data quality is huge. It reminds me of a talk I saw where someone said you aren't buying a solution, you're just buying a new set of problems to solve. The report shows the finish line, but not the swamp you have to cross to get there.
Do you think part of the issue is that these reports are written for the people buying the tools, not the people who have to live with them day to day?
Exactly, the high score on the chart is the beginning of the negotiation, not the end. You're paying for a demo-grade feature, which always breaks when you load your real production volume and schema history.
That three-month delay for custom work on "best-in-class" identity isn't an anomaly, it's the business model. The reports measure against a perfect, static spec, but real data is a moving target. They don't score how the system handles that loyalty system you acquired last year, where the customer ID format changed three times.
What you're seeing is the gap between marketing architecture and engineering architecture. One is built for slides, the other has to run at 2 AM when your primary key collision job fails. The operational load of maintaining their "best-in-class" match in the face of source drift is where you'll burn your team's capacity.
The "marketing architecture vs engineering architecture" split is exactly why my team stopped treating these evaluations as technical documents. They're financial risk assessments disguised as feature matrices.
Your loyalty system example is perfect, because it exposes the temporal problem these platforms ignore. A CDP that scores high on identity resolution today assumes your past data structures were rational and consistent. But mergers, legacy system sunsets, and that one intern who stored customer IDs in a VARCHAR(255) field with inconsistent padding create a historical debt no vendor's model can absorb without manual mapping. The three-month custom work isn't implementing a feature, it's paying them to write the documentation for your own data history that you never had.
The 2 AM primary key failure is the real test. When that alert goes off, you're not debugging Gartner's "Vision," you're debugging the thousands of implicit assumptions their match engine made about your data cleanliness. The operational load isn't a side effect, it's the primary cost they externalize onto your engineering team.
keep it simple