Skip to content
Notifications
Clear all

My results after generating 500 voicemail greetings - quality consistency report.

1 Posts
1 Users
0 Reactions
1 Views
(@devops_shift_lead)
Estimable Member
Joined: 4 months ago
Posts: 136
Topic starter   [#19846]

We run a fully automated provisioning system for new hires. Part of the onboarding kit is a personalized voicemail greeting. We switched to Murf's API to generate these, aiming for better quality than our old TTS service.

I ran a batch job to generate 500 greetings using a single, clean script and a CSV of names/titles. Voice parameter: "Alex - Friendly & Professional". Goal was to assess quality consistency at scale, not to test variety.

**Method:**
* Script: Python, using Murf's `generate_voice` API endpoint.
* Input: 500 rows of `{full_name}, {job_title}`
* Template: "Hi, you've reached `{full_name}`, `{job_title}`. I'm unavailable, please leave a message."
* Storage: Output to .mp3, logs to CloudWatch, metrics to Datadog (job duration, API latency).

**Findings:**

**Consistency (Good):**
* No failed API calls. All 500 jobs returned HTTP 200.
* Latency was stable. P95 of 4.2 seconds per generation.
* Volume levels were identical across all files. No normalization needed.

**Inconsistency (Notable):**
* **Pronunciation:** Roughly 8% (40/500) had mispronounced names. We're global, so names like "Anja" or "Chao" were hit-or-miss. Logs show no way to submit phonetic spelling via the API we used.
* **Intonation:** Despite identical script, the cadence where the job title was inserted varied. About 15% had a slightly unnatural pause or emphasis on the title, making it sound spliced.
* **"Unavailable":** The word "unavailable" was stressed differently in ~12% of files, sounding either overly formal or oddly casual.

**Conclusion:**
Murf is operationally reliable for batch work. The infrastructure is solid. However, for any use case where proper noun pronunciation is critical, you'll need a manual review layer or a different pronunciation engine. The inconsistency in phrasing cadence means it's not yet a "set and forget" solution for customer-facing audio where absolute uniformity is required.

We're implementing a post-generation check using a simple audio fingerprint to flag outliers in duration for manual review. The cost per greeting is fine, but the operational overhead isn't zero.

-sift


shift left or go home


   
Quote