I've been conducting a systematic analysis of Otter.ai's accuracy for a client considering it for technical meeting transcription. While the marketing claims are impressive, my controlled tests suggest a consistent overstatement of the word error rate (WER) by approximately 12-17%. This isn't just anecdotal; it's a measurable delta impacting utility.
My methodology:
* Used a set of 10 pre-recorded, high-fidelity audio samples covering clear monologue, technical discussions (with cloud/AI jargon), and meetings with mild crosstalk.
* Manually created a ground-truth transcript for each.
* Ran each sample through Otter.ai's web interface (default settings).
* Calculated WER using a standard script, comparing Otter's output to the ground truth.
The results showed Otter's effective accuracy averaged 83.5% on my mixed-sample set. However, when I fed the same audio into their system *after* a first-pass transcript was generated (simulating a user correcting errors), the platform's own "Accuracy Score" displayed was consistently 96-98%. This suggests their displayed score measures improvement from a baseline, not absolute accuracy against a ground truth—a critical distinction.
This has significant cost implications. If the baseline accuracy is lower than advertised, the "corrected" accuracy metric is misleading. The labor cost for human review and correction is the true TCO of any transcription service. An inflated accuracy score by 15% directly misrepresents the post-processing effort required.
Has anyone else performed a quantitative, ground-truth analysis? I'm particularly interested if others have controlled for domain-specific vocabulary, which seems to be a major source of errors in my tests (e.g., "EKS" becomes "X", "Azure Spot" becomes "Azure sport").
Less spend, more headroom.
No, you're definitely not the only one. That delta you measured aligns with what I've seen when teams try to bring automated transcripts into a CI/CD or post-mortem review process. The critical distinction you highlighted - measuring improvement from a baseline versus absolute accuracy against ground truth - is a classic vendor sleight-of-hand. It makes their correction tools look magically effective when in reality you're just doing the heavy lifting.
Your methodology is sound, but for technical jargon, I'd suggest adding a layer: track error rate by word complexity. In my experience, generic speech-to-text holds up until it hits specific pipeline tool names, error codes, or shorthand. That's where the real 15% gap lives, and it's the difference between a usable transcript and one that introduces new confusion.
Have you tested whether the "accuracy score" changes if you introduce errors intentionally on that first-pass transcript? I'd bet their system is just measuring edit distance from its own initial output, which is practically useless for your client's evaluation.
Speed up your build