Skip to content
Notifications
Clear all

My results: A/B test showed our AI avatar underperformed against a real person.

3 Posts
3 Users
0 Reactions
0 Views
(@devops_grunt_2024)
Estimable Member
Joined: 4 months ago
Posts: 148
Topic starter   [#8115]

Just spent the budget to run a proper A/B test on our new AI presenter from HeyGen. Pitted it against a video of our actual, boring, human PM delivering the same update.

Guess what? The "boring" human won. By a lot.

Key metric was viewer retention for a critical internal technical announcement. The AI avatar, despite the "futuristic" appeal, had a 40% higher drop-off rate in the first 30 seconds. Engagement metrics (questions asked in the follow-up survey) were also significantly lower for the AI version.

So we paid for a fancy tool to make a worse product. The team wanted to chase the shiny AI thing. I said just record a person. But no, we had to "innovate."

The config is fine, the lip-sync is okay, but it just doesn't land. There's a flatness to it that people tune out, even if they can't articulate why. Now we've got a library of AI videos that are less effective.

Stick with the boring, reliable human. Saves money and actually works.


If it ain't broke, don't 'upgrade' it.


   
Quote
(@ci_cd_crusader_v2)
Estimable Member
Joined: 3 months ago
Posts: 135
 

Your story mirrors what happens when teams treat CI/CD like a shiny toy rack. They buy the bloated SaaS platform with all the AI-powered "insights" instead of fixing their flaky test suite.

You "innovated" and got a worse product with more moving parts. I'm not surprised. The uncanny valley of engagement isn't just for faces. There's a flatness to over-engineered processes, too. People, and their boring, reliable workflows, often just work better.

The real cost isn't the HeyGen subscription. It's the team's time now tied to maintaining that "futuristic" content pipeline when a quick recording would do. Sounds familiar.


null


   
ReplyQuote
(@henry)
Estimable Member
Joined: 1 week ago
Posts: 79
 

You're spot on about the uncanny valley for engagement. It's not just about looks - it's about connection.

We saw the same thing with AI-narrated training modules. Completion rates tanked. Even with perfect audio, people disengage when they can't pick up on those tiny human cues, like a slight pause before an important point.

Curious, did you track any sentiment? We found the AI version scored lower on "trustworthiness," which for an internal tech update is everything. Might help your case for the next "shiny thing" discussion.


Cheers, Henry


   
ReplyQuote