I've been thinking about this a lot while evaluating new marketing automation platforms at work. We're in procurement for a new CDP integration, and every vendor demo is a perfectly curated success story. But we all know that's not the whole picture.
My hot take: we should start requiring vendors to submit reproducible failure logs as part of the RFP process. Not just a list of known bugs, but specific, documented instances where their AI or automation features failed under realistic conditions. For example, a prompt that caused their content assistant to hallucinate non-existent API endpoints, or a specific A/B test configuration that their analytics engine misinterpreted.
I ran into something like this last month with a popular email platform. Their "smart send time optimization" kept scheduling emails for 2 AM for a segment of business customers. The sales rep called it an "edge case." Our logs showed it happened every Tuesday. If we'd seen that failure case documented upfront, we could have asked how they fixed it, or at least budgeted for the workaround.
What would that look like in practice? Maybe a standardized template: the user prompt or configuration, the incorrect output, the correct/expected behavior, and the conditions that triggered it. It would move us from "look what it can do" to "here's how it breaks, and here's how we handle it."
I'm curious if anyone else has tried to formalize something like this during procurement, especially for tools heavy on AI features. It feels like the only way to get past the marketing gloss and understand the actual day-to-day friction.
Yes! This is brilliant. I've started asking for failure case studies in security reviews, but never thought to apply it to procurement.
The trick is making it about process, not shame. Frame it as, "Show us how you handle a failure log from discovery to fix." Their response tells you everything about support and product maturity.
Good luck with the CDP search. The 2 AM email story is painful.
Let's build better workflows.