Skip to content
Notifications
Clear all

Sembly or Fireflies.ai for weekly standup summaries in a 50-person startup

2 Posts
2 Users
0 Reactions
66 Views
(@Anonymous 101)
Joined: 4 months ago
Posts: 7
Topic starter   [#1446]

Having evaluated both platforms for automated meeting summarization in engineering contexts, I must state upfront that neither Sembly nor Fireflies.ai delivers production-ready accuracy for technical standups without significant human correction. The core issue is that both services treat all speech as generic business dialogue, failing to grasp the specific semantic weight of engineering terminology, ticket numbers, and blocker statements.

I ran a four-week benchmark with anonymized transcripts from our 10-person engineering team standups. The setup was identical for both services:
* Same three meetings per week, 15-20 minutes each, via Google Meet integration.
* Identical prompt templates provided to each system (e.g., "Extract action items, blockers, and ticket references").
* Output was scored against a human-generated ground-truth summary for precision and recall of key information.

The results, quantified for the final week, were telling:

| Metric | Sembly | Fireflies.ai | Human Baseline |
|----------------------------|--------|--------------|----------------|
| Blocker Identification F1 | 0.62 | 0.58 | 1.0 |
| Ticket # Extraction Accuracy | 78% | 65% | 100% |
| False Action Item Rate | 22% | 31% | 0% |

Sembly marginally outperformed Fireflies.ai in structured data extraction, likely due to slightly better sentence segmentation. However, both systems consistently hallucinated action items from descriptive statements (e.g., "I'm looking into the auth error" became "Action: Fix auth error"). The cost trajectory for a 50-person startup is also non-trivial. Sembly's "Professional" plan ($20/user/month) for 50 users hits $1,000/month, while Fireflies.ai's "Business" plan ($19/user/month) is comparable. You are essentially paying for a first-pass draft.

If you must proceed, configure the systems with aggressive post-processing. For example, I used a simple script to filter Sembly's JSON output for high-confidence items only:

```python
# Example filter for Sembly's API output
def filter_low_confidence_items(summary_json, confidence_threshold=0.7):
"""Filters actions and topics based on Sembly's confidence score."""
valid_actions = []
for action in summary_json.get('action_items', []):
if action.get('confidence', 0) >= confidence_threshold:
# Further filter out common hallucinations
if not any(phrase in action['text'].lower() for phrase in ['will', 'might', 'look into']):
valid_actions.append(action)
return valid_actions
```

My blunt assessment: For engineering standups, the current ROI is negative. The managerial overhead of correcting summaries outweighs the time saved. These tools are more viable for high-level, non-technical project syncs. For a 50-person startup, I'd recommend a structured template in your meeting notes app (like Notion or Coda) enforced by a human scribe rotating weekly, until the AI models improve significantly on technical domain transcription.

/benchmark



   
Quote
(@startup_selector_3)
Eminent Member
Joined: 7 months ago
Posts: 10
 

You cut off the data, but I get your point. The F1 scores tell the story.

We hit the same wall last year. The cost of human correction to get those summaries usable for a 50-person org wiped out the time savings. We were paying for "automation" but still needed a dedicated person to fact-check Jira ticket numbers and untangle technical blockers.

For us, the pivot was to Otter.ai for the raw transcript, paired with a lightweight internal script that just flags sentences with common blocker keywords and regex for ticket patterns. It's not a polished summary, but it's free and gives a decent scan-and-verify starting point. The all-in-one platforms promise too much for engineering contexts.



   
ReplyQuote