Skip to content
Notifications
Clear all

Hot take: Otter's meeting summaries are getting less accurate, not more.

2 Posts
2 Users
0 Reactions
2 Views
(@davidr)
Reputable Member
Joined: 2 weeks ago
Posts: 170
Topic starter   [#22562]

I've been a paying Otter Business customer for three years, primarily using it to transcribe and summarize technical planning meetings and post-mortems. The core value proposition for me was always the accuracy of the *summary*, not just the raw transcript. Over the last six to nine months, I've observed a marked and consistent decline in the quality and accuracy of these automated summaries. This isn't a minor gripe; it's a fundamental degradation of the product's utility.

My hypothesis is that as Otter has pushed more "AI features" like action item extraction and generic "AI Chat," the core summarization logic has suffered. The summaries are becoming more generic, missing critical technical nuance, and increasingly hallucinating points that were never discussed.

Here's a concrete example from a data pipeline design session last week. The discussion centered on the trade-offs between Apache Iceberg and Delta Lake in a new S3 data lake, with specific debate on merge-on-read vs. copy-on-write performance for our write pattern.

**What was discussed:**
* The need for schema evolution support, with a focus on adding nullable columns.
* Detailed benchmarking of `MERGE` operation performance in our test environment with ~1TB of data.
* Concern over AWS Glue's Spark runtime compatibility with Iceberg's Java API.

**What Otter's "AI Summary" produced:**
* "The team talked about data lakes."
* "A decision was made to use either Iceberg or Delta."
* "Performance was considered important." (No specifics)
* "Action Item: David to look into cloud storage options." (This was never stated or assigned.)

This is useless. The summary failed to capture the *actual* decision point (merge-on-read for our frequent small writes) and invented an action item. I am now forced to re-read the entire transcript, which defeats the purpose.

I've attempted to mitigate this with custom vocabulary (adding "Iceberg," "Delta Lake," "Glue," etc.), but it doesn't seem to influence the summarization layer, only the transcription. The workflow is now broken.

**My questions to the community:**
* Is this a widespread experience, or am I an outlier? Are others in technical fields seeing this?
* Has anyone found a workable configuration or prompt technique to force more accurate, detailed summaries?
* If this is the new normal, what alternatives are you evaluating? I've started logging my own benchmarks for local LLM summarization of exported transcripts, but the integration overhead is significant.

The product seems to be optimizing for "look what our AI can do!" over "our AI reliably delivers accurate, context-aware summaries." For a data-driven professional, the former is a party trick; the latter is a tool that saves hours per week.

—davidr


—davidr


   
Quote
(@gracyj)
Estimable Member
Joined: 2 weeks ago
Posts: 91
 

Completely agree, and it's frustrating. The shift from a focused summary tool to a general "AI assistant" feels like it's hurting its original job. I've seen the same thing in client onboarding calls, where the summary now misses key concerns about implementation timelines but confidently invents generic next steps.

I wonder if their training data is getting noisier. When you add features for every type of meeting, maybe the model loses its edge on technical or niche discussions like your data pipeline example. Have you tried feeding it the transcript as a custom vocabulary or prompt first? That sometimes helps me claw back some accuracy, but it's extra work that shouldn't be necessary.


Happy customers, happy life.


   
ReplyQuote