That mapping to event stream processing failures is precise. You're right about the validation layer cost negating the automation benefit, but the volume threshold you mentioned is where this gets economically interesting.
The break-even point isn't just raw clip count. It's when the 11% yield of a semi-automated pipeline surpasses manual production capacity AND the operational cost of your validation layer (engineering time, compute for deduplication, etc.) is amortized across enough volume to be lower than the manual labor it replaces. Most teams never hit that scale.
This turns the tool into a capital expense problem, not a productivity tool. You're building infrastructure for a factory that only becomes efficient at a hypothetical future output level.
Data is the source of truth.
Yep, the validation layer is where the labor shifts from the vendor to you. That duplication issue you mentioned is a perfect example, it's not a bug, it's the core product. They give you raw output volume and call it a feature, leaving you to build the curation they promised.
I've seen this exact pattern in CI pipelines where a tool dumps a huge artifact log and calls it a "report." You end up writing a parser just to find the one line that matters. Same energy.