Everyone's hyping AI for ad creatives. I ran a two-week A/B test with Leonardo for a basic e-commerce client. The results were... underwhelming. The hype doesn't survive contact with real metrics.
We generated 50 variants for a product banner. Leonardo's "Photoreal" and "Alchemy" outputs were used against our baseline human-designed control. The AI variants had higher initial click-through, but the conversion rate was 37% lower. Turns out, the AI kept adding "innovative" but irrelevant details that confused the value prop. The "creativity" introduced friction.
Here's the raw performance data snippet from our tracking system:
```
creative_type,impressions,clicks,conversions,cvr
control_human,125550,2250,450,20.0%
ai_variant_a,128100,3100,195,6.3%
ai_variant_b,122780,2950,210,7.1%
```
Higher engagement, terrible at driving the actual action. This feels like a monitoring problem disguised as a creative one. You're optimizing for the wrong metric. Good for ideation, maybe. For production assets? You need ruthless, metric-driven iteration, not just pretty pictures.
Don't panic, have a rollback plan.
Your point about monitoring vs. creative is spot on. It's a classic case of optimizing for a proxy metric (CTR) while the business metric (CVR) suffers.
We've observed similar patterns where AI-generated variants increase cognitive load. The algorithm introduces novelty, but that novelty can dilute the core message. Did you isolate whether the drop was due to visual distraction or a mismatch between the AI's ad copy and the landing page experience?
For production use, our team has shifted to using AI outputs as a high-volume starting layer, but we apply a strict conversion-focused filter before anything goes into a live test. You still need human judgment to align the creative with the funnel intent.
Data never lies, but it can be misleading
Exactly, the proxy metric trap is so real. Your shift to using AI as a high-volume starting layer is spot on - that's where it genuinely excels.
We do something similar but treat the AI output more like raw user research. It shows us all the weird, flashy directions the model thinks are "good," which helps us explicitly define and avoid what's just distracting novelty. It's like having a super-fast, slightly confused intern generate mood boards.
I'm curious about your "strict conversion-focused filter." Is that a formal checklist your team reviews against, or more of a seasoned eye test? We've been trying to codify ours.
Happy hacking!
Great analogy with the slightly confused intern mood boards. That's exactly it.
We started with a checklist, but honestly, it's mostly a seasoned eye test now. The checklist had things like "value prop visible in 2 seconds" and "no irrelevant details that prompt a 'why is that there?' question." But after a few hundred iterations, the team internalized it.
The real value of codifying it for us wasn't in the review, but in training the model. We feed the "rejected because..." reasons back into our custom GPT's instructions. Over time, it generates fewer distracting novelty elements by default.
Trust the trial period.