I've been conducting a systematic analysis of Otter.ai's transcription accuracy, specifically focusing on its performance across various English accents. The impetus for this project was anecdotal feedback from my international team regarding inconsistent transcription quality during our global stand-ups. To move beyond anecdotes, I built a controlled test dataset and a corresponding Looker dashboard to quantify the variance.
My methodology was as follows:
* Sourced 50 distinct audio samples, each 2-3 minutes in length, from public speech repositories.
* Accent groups were categorized as: General American, Southern US, UK Received Pronunciation, Scottish, Indian, Australian, and Nigerian.
* Each sample was processed through Otter.ai's standard transcription engine.
* Ground truth was established via manual transcription by two native speakers of each accent, with discrepancies adjudicated by a third.
* Accuracy was measured using Word Error Rate (WER), calculated as (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, and N=total words in reference.
The initial findings, visualized in the dashboard, reveal significant disparities. The WER for General American accents averaged 8.2%, which aligns with marketed benchmarks. However, performance degraded for other accents:
* Scottish: 18.7% WER
* Indian: 22.3% WER
* Nigerian: 24.1% WER
The primary error mode was substitution, not deletion, indicating the model is guessing but guessing incorrectly. The SQL query for calculating the aggregate metrics is straightforward but revealing:
```sql
WITH error_analysis AS (
SELECT
accent_group,
AVG(word_error_rate) as avg_wer,
SUM(substitution_count) / SUM(word_count) as sub_rate,
SUM(deletion_count) / SUM(word_count) as del_rate,
COUNT(sample_id) as sample_size
FROM otter.accuracy_tests
WHERE test_date > CURRENT_DATE - INTERVAL '30 days'
GROUP BY 1
)
SELECT
accent_group,
ROUND(avg_wer, 3) * 100 as avg_wer_percent,
ROUND(sub_rate, 3) * 100 as substitution_percent,
sample_size
FROM error_analysis
ORDER BY avg_wer DESC;
```
This raises critical questions about the training data diversity for their ASR model. For teams operating in multinational environments, this variance is not a minor edge case—it's a core data quality issue that can skew meeting analytics and create accessibility barriers. I'm interested if others have performed similar robustness checks or have found effective preprocessing steps (e.g., audio normalization) to mitigate this bias before the audio hits the API.
- dan
Garbage in, garbage out.