Great point on the data consistency domino effect. It's a real problem when speaker labels go sideways early in the pipeline.
Your pre-process idea is solid, but as others noted, it's tough to enforce. What I've seen work as a practical middle ground is tagging those low-confidence speaker switches in Datadog or a similar system and having a second-tier alert fire. That way, your review team isn't sifting through entire transcripts - they're just jumping to the 2-3 segments per call where the diarization score dropped below a threshold. It adds a delay, but it's a targeted one.
Curious - have you tried running the audio through a cheap, separate speaker diarization service first (like pyannote) just to get an initial speaker count and rough segments, then feeding *that* metadata to Descript? Sometimes giving the main tool a hint goes a long way.
Dashboards or it didn't happen.
Your point about the secondary check collapsing similar speakers based on a known roster is a sound practical mitigation. It's a classic trade-off between precision and recall: you'll inevitably merge a few distinct speakers if they're acoustically close, but you'll prevent a larger fragmentation error across the transcript.
I've implemented a similar stage, and the real challenge is tuning that confidence threshold. It's not static. The similarity measure between "Speaker 2" and "Speaker 3" segments needs to be evaluated relative to the within-speaker variance for each person in your roster, not just a global cosine similarity cutoff. If Susan's voice is highly consistent but Sarah's varies with fatigue or mic position, you need adaptive thresholds, which adds more complexity to that lightweight step.
Have you found a particular feature representation for the voice snippets in that staging queue that works reliably, like a mean pooled layer from a small embedding model, or are you working directly with the vendor's proprietary speaker vectors?