Good point. Comparing their ability to guess from perfect audio only tells us about their internal dictionaries, not their real-world resilience.
For simulating degradation, I've seen folks get decent results by running clean audio through a low-bitrate Opus encoder, maybe with some simulated packet loss. That mimics the codec path. But you're right that replicating Teams' specific noise suppression is trickier. I wonder if the real test is just feeding them actual archived Teams recordings and comparing.
~Harry
I like your approach, especially the use of a senior engineer for the ground truth transcript. That's key for judging intent, not just raw word accuracy.
Your mention of high-quality source audio is interesting though. It means you're mainly stress-testing their language models' internal knowledge of technical terms, which is useful, but it sort of skips the most common failure mode in a real workflow. Most of us are dealing with the audio after Teams or Zoom has had its way with it.
I'd be really curious if the gap between Granola and Tactiq widens or narrows when you feed them a version of that same audio after it's been run through a low-bitrate Opus encode to simulate a typical call stream.
Your methodology's use of high-quality source audio is a controlled starting point, but it fundamentally tests language model bias, not the acoustic model's resilience to degraded input. The absolute accuracy numbers you get from this clean feed will be misleadingly high and likely compress the practical difference between the services.
When you process that same audio through a low-bitrate Opus codec to simulate a real conferencing pipeline, you'll likely see a divergence. A service with a stronger acoustic model will maintain a higher proportion of its clean-audio accuracy, while one reliant on pristine input for jargon recognition will degrade more sharply. The error profile will also shift from plausible substitutions to garbled phonemes, which changes the operational risk.
The real test isn't which service understands "EC2" from a perfect recording, but which one can disambiguate "EKS" from "X" when the high-frequency consonant data is lost to compression. Your current setup answers the first question, not the second.
Data never lies.