I've been experimenting with Resemble AI's voice cloning for a data pipeline project. I generated 50 hours of synthetic speech from text using my custom voice model to test its robustness.
I used the Python `pesq` and `pystoi` libraries to compare the synthetic output to my original recordings. The average PESQ score was 2.8, and the average STOI was 0.86. For my use case, this intelligibility is okay, but the naturalness feels a bit off.
Has anyone else run similar objective tests? I'm curious how these scores compare to other services or if there are better metrics for long-form synthetic speech. Also, did you find a "sweet spot" for training data amount?
Those scores line up with what I've seen when testing synthetic voices against those exact metrics. PESQ around 2.8 screams "mechanically intelligible but unnatural," which is exactly what you described. It's a decent benchmark for a pipeline, but your ear is picking up what the numbers can't fully capture.
You asked about a "sweet spot" for training data. I'm skeptical of the idea that just throwing more hours at it fixes the naturalness problem. I've seen models trained on 20 hours of clean, phonetically-diverse data outperform ones trained on 100 hours of varied-quality, redundant material. The quality and breadth of your 50 hours matters more than the raw quantity.
Ever look at DNSMOS or similar non-intrusive metrics for your longer-form stuff? Less of a headache than running reference-based tests on hours of audio, though you trade some accuracy.
null