I've been using Cartesia's API for a few months now, primarily to synthesize voiceovers for internal system alerts, and I've been generally impressed with the quality and latency. However, as a team, we've had some debates about whether the emotional range in certain voices has been consistent across updates. To move beyond anecdotal feedback, I built a small script to programmatically fetch our historical synthesis requests, analyze the sentiment of the generated audio, and plot the trends over time.
The core idea is simple: use Cartesia's API to get a list of past jobs, fetch the audio, run it through a speech-to-text service (I used Whisper via a local API for privacy), and then perform sentiment analysis on the transcribed text. This gives us a rough metric to see if the perceived "emotional delivery" of a given voice model has drifted. The script uses Cartesia's Python SDK and plots the results.
Here's the main visualization script. You'll need to set your `CARTERSIA_API_KEY` and configure your own Whisper endpoint.
```python
import pandas as pd
import matplotlib.pyplot as plt
from cartesia import Cartesia
from datetime import datetime
import requests # For calling your Whisper & sentiment endpoints
# Initialize
cartesia = Cartesia(api_key="your_key_here")
sentiment_scores = []
# Fetch recent jobs (adjust limit as needed)
try:
jobs = cartesia.jobs.list(limit=100)
except Exception as e:
print(f"Error fetching jobs: {e}")
jobs = []
for job in jobs:
if job.status == 'completed' and job.audio_url:
# 1. Transcribe audio (pseudo-code for your Whisper API)
transcription = transcribe_audio(job.audio_url)
# 2. Analyze sentiment (simple API call, returns polarity)
sentiment = analyze_sentiment(transcription)
# 3. Store with timestamp
sentiment_scores.append({
'date': job.created_at,
'model': job.model,
'voice_id': job.voice_id,
'score': sentiment,
'text_preview': job.input_text[:50]
})
# Create DataFrame and plot
df = pd.DataFrame(sentiment_scores)
df['date'] = pd.to_datetime(df['date'])
plt.figure(figsize=(12, 6))
for voice in df['voice_id'].unique():
voice_data = df[df['voice_id'] == voice]
plt.plot(voice_data['date'], voice_data['score'], marker='o', label=voice, linestyle='-')
plt.title('Sentiment Polarity Trend per Voice Model (Cartesia Synthesis)')
plt.xlabel('Synthesis Date')
plt.ylabel('Sentiment Polarity Score')
plt.legend(title='Voice ID')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
```
A few important caveats and insights from my run:
* This measures sentiment from the *transcribed text*, not the audio prosody itself. It's a proxy metric, but it surfaced an interesting pattern for one of our frequently used "neutral" voices where the sentiment score became slightly more positive over time for identical input text, which aligned with our team's subjective feeling.
* You'll need to implement the `transcribe_audio` and `analyze_sentiment` functions. I used a local text analysis model to avoid sending data externally.
* Rate limiting and job history retention on your Cartesia plan will dictate how much data you can analyze.
* This isn't a critique of Cartesia's service—it's more about creating an observability layer for our own usage, much like we monitor our microservices. It helps us decide when to re-evaluate voice parameters or report subtle shifts.
I'm curious if others in the community have built similar tooling for tracking synthesis quality or have thoughts on better metrics for vocal emotion consistency over time. The script is a starting point and could be extended to track latency, cost per job, or even integrate with a service mesh for broader API performance insights.
—Josh
Design for failure.