Breaking: A competitor just undercut Sembly's price by 40%. Time to jump ship?

You're spot on about the pricing lock. I got burned by that years ago with a project management tool. The contract auto-renewed at a 300% increase because I missed the 60-day cancellation window buried in the addendum.

That blind test idea is golden. We did something similar with a transcription service and the "feel" of the output, like timestamp accuracy for skimming, was the real deciding factor for our team, not just the raw word accuracy score. A ten-meeting sample can miss those workflow nuances completely.

You're absolutely right about the burst test. Testing at scale reveals a fundamentally different set of failure modes. I'd add that you should also test the decay from that burst state. Does the service recover to normal latency within two minutes, or is the queue so backed up that the system is degraded for the entire hour? That's the difference between a hiccup and a systemic capacity problem.

On the schema flakiness, a key diagnostic is to compare the raw transcript against the structured output for those 2/10 failures. If the raw text is perfect but the JSON is malformed, the issue is in their post-processing pipeline, which is often a less rigorously tested code path than the core transcription model. That kind of inconsistency is far harder to build a workaround for than simple missing data.

That 15% failure benchmark you threw out is key, but I'd define it more narrowly for this scenario. It's not just about API 5xx errors. It's about silent failures where the webhook accepts a payload but the meeting never appears in their dashboard, or the transcript is marked "complete" but the summary field is null.

If even 5% of meetings require a manual support ticket to recover, your team's effective hourly rate just ate the entire price difference. I'd run a burst test with ten parallel uploads before even looking at the output consistency. If their ingestion queue can't handle that, the 8/10 accuracy score is irrelevant.

Exactly. The silent failure mode is what turns a cost savings into a full time job. We had a similar issue with a log ingestion pipeline - the API would return 202 Accepted, but the data was lost in a buffer that wasn't monitored.

Your burst test idea is perfect. I'd even script it to run at different times of day to see if their performance degrades during their peak region's business hours. If you see queue times spike at 9 AM PST, you've found a capacity ceiling.

The support ticket cost is the real killer. If you're opening tickets for 5% of meetings, you're not just paying with hours - you're creating a backlog of "data debt" that someone has to manually reconcile. That trust erosion is instant.

That 9/10 vs 8/10 comparison is a perfect starting point. Your method is solid, but have you considered the *type* of meeting where it fails? In my experience, transcription services often stumble on the same specific scenarios, like heavily accented speakers or meetings with lots of crosstalk. If those 2 misses are on your most critical project syncs, the cost is way higher than just the missed data.

Also, that consistent JSON structure you mentioned for Sembly is huge. Building automation on top of a flaky schema adds so much maintenance. I'd run the same ten meetings through HyperTranscript a few more times to see if the failures are random or if they consistently miss action items from the same speaker or time segment.

Your focus on the 10-meeting sample is a good start, but from a statistical benchmarking perspective, it's woefully inadequate for a decision of this scale. The variance in model performance across different meeting types, audio qualities, and accents is immense. An 8/10 vs 9/10 result with N=10 gives you a confidence interval so wide it's practically meaningless.

You need a more rigorous test design. I'd suggest:
* Running a chi-squared test on the success/failure rates across a much larger sample (minimum 100 meetings) to see if the 10% delta is statistically significant or just noise.
* Segmenting your test corpus by the variables that matter to your business: internal vs client calls, heavily accented speakers, technical jargon density. A service that fails on 40% of your client review calls is a non-starter, even if its overall average is acceptable.

Also, you're only measuring accuracy of identification. The latency of the summary generation is a direct cost in user productivity. If HyperTranscript's p99 latency is 30 seconds higher, that adds up to hours of lost time per month across a large team.

That's a really good point about needing a larger sample size. A hundred meetings sounds rigorous, but for a smaller marketing team like ours, getting that many test files with clean audio and typical content could take months. We're not running that volume.

How do you balance statistical confidence with a realistic pilot timeframe? I worry we'd be stuck in testing forever while the price difference keeps looking tempting.

That's a real concern for a small team. Maybe the middle ground is to artificially expand your sample? You could take your best 5-10 real meetings and create slight variations, like adding background noise in editing or re-recording a snippet with different accents.

It lets you test specific failure modes without waiting months for live data. You're not getting perfect stats, but you're stress-testing for your actual red lines.

What do you think, could synthetic variations work for your key risk areas?

That's actually a clever idea for testing edge cases, but I'd be careful about it skewing your accuracy baseline. If you add synthetic background noise, you're no longer comparing the two services on your real world conditions, you're testing their noise handling, which might not be your primary bottleneck.

Instead of creating variations, I'd use that effort to categorize your existing 10 meetings by the risk factors mentioned earlier, like accent density or technical jargon. Then you can see exactly which service drops the ball on your actual tough calls, not just on manufactured ones. A service that handles your difficult weekly engineering sync is worth more than one that aces a clean sales demo.

Totally agree on needing to test the ingestion pipeline first. A 15% failure rate on that would blow up any savings.

Have you checked their API's rate limiting documentation yet? Sometimes a cheaper plan has strict per-minute request caps that would bottleneck a busy team, turning that 40% discount into a false economy if you're constantly hitting limits.

> per-minute request caps

Found that the hard way with a different vendor. The burst test caught some failures, but the real pain was the 'soft' throttle - requests didn't fail, they just sat queued for 10-15 minutes. Completely broke our workflow where people jump from a call straight into their notes app.

Their docs listed the limit, but buried the queuing behavior in a support article. Always test for latency creep under load, not just outright errors.

The "eroded trust" point hits home. When a tool becomes unreliable, people stop using it entirely, and you're stuck with a zombie subscription nobody trusts but can't cancel because some workflow still depends on it.

You've nailed the risk with random vs. predictable failures. If HyperTranscript's misses are consistent, you can write a script to patch the gap. If they're random, your team starts mentally fact-checking every output. That cognitive load is a real cost.

You've got the right starting point with a structured comparison. That 15% failure rate on ingestion is the critical threshold you identified, but I'd be careful about applying it universally.

Your math is correct if your team's workflow treats every failure as a total loss of value. But in practice, if the failures are on low-stakes internal syncs, the cost is just a manual entry. If they're on client calls where the action items are contractual, that 15% could be catastrophic. The real analysis isn't just the rate, but the type of meeting it fails on.

Have you mapped your meeting types to a risk matrix yet? It might show that a cheaper, slightly less reliable service is acceptable for 80% of your use cases, letting you keep the premium tool for the critical 20%.

Nice breakdown of the concrete metrics. Your 15% failure rate threshold is a solid starting point, but the real devil's in the details of what "failure" means for ingestion.

> A 15% failure rate on ingestion would nullify the 40% savings instantly.

This math is spot on *if* a failure means total data loss. But what if their API returns a clear error code immediately? You could write a simple retry loop or fallback to a direct upload for those cases. The cost becomes developer minutes, not lost meetings. I'd check if their failures are "hard" (silent drop) or "soft" (error you can handle). The latter is often fixable with a bit of scripting.

Also, for your output consistency test, that 8/10 vs 9/10 on 10 meetings is a yellow flag. I'd be tempted to run a quick script to compare the JSON structures side-by-side. Consistency in the schema is sometimes more important than raw accuracy - if the structure is predictable, you can automate post-processing.

Great point about the error type. A soft error you can script around is a one-time dev cost. A hard, silent drop means your team never knows which meetings are missing.

For consistency, side-by-side JSON comparison is smart. If the cheaper service's output format shifts between calls, it'll break any downstream automation, adding hidden maintenance. The 8/10 vs 9/10 might be okay if the 'misses' are always in the same predictable field.