We ran a 30-day Kling POC for automated video generation on our landing pages. The results were clear and didn't meet our success criteria.
Key metrics vs. our baseline:
* **Completion rate:** 12% lower
* **Avg. time on page:** Down by 23 seconds
* **Conversion rate:** No statistically significant change (p-value > 0.1)
* **Dev resources consumed:** 18 hours for integration and bug fixes
The tool works, but it didn't improve our core goals. It added friction without lifting conversions. The data is neutral—it just shows the experiment didn't win.
I need to present this as a learning, not a failure. How have you structured these conversations? Focus on the objective gates we set and what we learned about our users.
Optimize or die.
Focusing on the objective gates is exactly the right approach. You had a hypothesis, you defined clear metrics for success before the POC started, and the data shows it didn't meet them. That's the opposite of sounding negative, it's just following the scientific method.
One thing I'd add is to explicitly call out the learning about user friction. The drop in completion rate and time on page tells a story about user experience that's more valuable than just a "pass/fail" on conversion. It suggests the video format, or its implementation, was disruptive. That's a concrete insight you can apply to future page designs.
Also, quantify the 18 dev hours as an opportunity cost. Frame it as, "We invested this time to learn X, which saved us from a potential full-scale rollout that would have cost Y." That positions the POC as a prudent, low-risk investment in research.
null
> positions the POC as a prudent, low-risk investment in research
Sure, if you want to pretend 18 hours of dev time on a video tool that made your site worse is "research" instead of a poorly scoped experiment. The opportunity cost framing works if your manager buys that line. Mine wouldn't. She'd see a failed POC and ask why we didn't kill it after week one when the metrics started trending down.
The "scientific method" analogy is nice in theory. But in practice, the moment you set up a pass/fail gate, you're committing to run the full test regardless of early signals. That's not science, it's decision paralysis. You could have learned the same thing about user friction by running a one-week A/B test with a smaller sample and saved that 18 hours.
I'd rather present the data as: the tool's output didn't justify the integration overhead, full stop. No need to dress it up as learning unless you're actually going to apply that learning differently next time. Which, let's be honest, most teams won't.
Don't panic, have a rollback plan.
I agree with framing it around your predefined gates, but the key is making your analysis actionable. You've correctly identified that the video added friction, but you should drill into why. Did you capture any qualitative feedback or session replay data to explain that 23-second drop in time on page? Was it a player loading delay, or did users simply skip past the video?
Pairing those behavioral metrics with the technical integration cost gives you a complete cost/benefit picture. For a truly data-driven conclusion, I'd map the 18-hour integration effort against the lost user-seconds from the drop in engagement. That calculation often shows the scale of the negative impact more starkly than the percentages alone.
This turns the conversation from "the POC failed" to "we validated that automated video, as implemented, degrades our core user journey, and here's the evidence guiding our next experiment."