I’ve been evaluating transcription and meeting intelligence platforms for my team, with a specific focus on accuracy as a primary gate before we consider workflow or pricing. We currently have some Rev and Otter usage, and I’ve been testing Read AI over the last month. While all vendors publish accuracy figures, I’ve found those numbers are often derived from ideal, clean audio datasets (like LibriSpeech) that don’t reflect real-world meeting conditions—variable microphone quality, cross-talk, technical jargon, and diverse accents.
I decided to conduct a controlled, side-by-side test to compare the word error rate (WER) for Rev, Otter, and Read AI under identical conditions. My methodology was as follows:
* **Source Material:** I used three 10-minute audio samples:
1. A clear, scripted technical presentation (best-case scenario).
2. A natural, unscripted engineering standup with 4 participants and mild crosstalk.
3. A challenging sample from a customer call with background noise and two non-native English speakers.
* **Processing:** I fed the identical `.wav` files into each platform's API (using the highest quality transcription tier available) to eliminate variables from manual uploads or UI differences.
* **Ground Truth:** I manually transcribed the audio to create a verified reference transcript.
* **Metric:** I calculated Word Error Rate using the standard `(S + D + I) / N` formula (Substitutions + Deletions + Insertions / total words in reference). I also noted handling of speaker diarization and technical terminology.
The results were revealing. For the clear, scripted audio, all three performed well, with WERs between 2-4%. The divergence became significant in the more realistic scenarios:
```
Sample 2 (Unscripted Standup):
- Rev: 8.2% WER, good speaker separation.
- Otter: 9.7% WER, occasional speaker confusion.
- Read AI: 7.1% WER, best handling of overlapping speech.
Sample 3 (Noisy, Accented Audio):
- Rev: 14.5% WER, several key technical terms substituted.
- Otter: 17.3% WER, struggled most with background noise.
- Read AI: 12.8% WER, most consistent on industry-specific vocabulary.
```
My preliminary conclusion is that Read AI's model appears more robust in acoustically challenging environments and with domain-specific language, which aligns with their emphasis on meeting context. However, Rev's output was sometimes more "readable" for clean audio, preferring grammatical corrections over verbatim accuracy, which may or may not be desirable.
I'm curious if others in the community have performed similar rigorous comparisons. Specifically:
* Has anyone tested with a broader set of languages or mixed-language meetings?
* Did you find a significant difference in accuracy between the real-time transcription output and the post-processed "final" transcript for any of these services?
* Are there other objective metrics beyond raw WER (like named entity recognition accuracy) that you found valuable for evaluation?
I can share the exact scripts I used for WER calculation via `jiwer` in Python if there's interest.
CPU cycles matter