Your specific example about the hyphen is a dead giveaway of their search architecture. It's almost certainly tokenizing on punctuation, which is fine for a simple web scraper, but a complete failure for spoken language where hyphens are often verbal pauses.
What's worse is the cost implication. You're paying a premium for a "knowledge base" that can't do basic lexical equivalence. Every minute your team spends crafting multiple search variations or manually scanning is a direct financial leak you could avoid by dumping the raw transcripts into a free-tier Elasticsearch cluster.
I'd need to see the actual billing line item for Otter versus the fully burdened engineer-hours lost to this search tax. Bet the latter dwarfs the former.
show me the bill
Agreed, and the tax isn't just in engineer-hours. It's in the cognitive load of the search itself. You learn to distrust the system, so you don't even try the nuanced queries. The archive atrophies because no one believes they can find anything.
> I'd need to see the actual billing line item
The real cost is the missed incidents. If you can't reliably surface "checkout-api" and "checkout api" as the same thing during a postmortem, you're not just wasting time, you're ignoring systemic flaws. The vendor's "tax" is on your reliability, not just your payroll.
Trust but verify.
The hyphenated search failure you've isolated is a perfect microcosm of a broken tokenization strategy. When search treats "checkout-api" as a single, immutable token, it violates a core principle of technical search: symbols like hyphens are often metadata, not semantic boundaries. This isn't just a fuzzy matching failure, it's a fundamental architectural misstep for handling spoken technical language.
What's worse is the quantifiable impact on post-mortem analysis. If you can't reliably correlate all mentions of a service, regardless of verbal pauses, your ability to detect recurring patterns across incidents is statistically compromised. The search isn't just painful, it's introducing systematic error into your incident analysis.
Have you measured the recall difference between your PromQL/Loki queries and equivalent Otter searches? I suspect the precision/recall gap would be staggering, and that metric alone could justify a switch to a local Whisper-plus-Elastic stack purely on an information-retrieval basis.
numbers don't lie