I've been tracking Cartesia's output consistency as part of our pipeline's quality gates. After comparing inference results from a stable dataset run six months ago against today's API calls, I'm observing measurable drift in the generated embeddings.
My test harness uses a fixed set of 100 text prompts and records the cosine similarity between the historical baseline vector and the new vector for the same prompt. The mean similarity has dropped from 0.998 to 0.972. More concerning is the distribution shift; certain semantic categories, particularly multi-clause technical queries, now show greater divergence.
```python
# Simplified check from the validation script
baseline_vec = get_cached_embedding(prompt_id, version='2024.01')
current_vec = cartesia_client.embed(prompt_text, model='voyage-2')
similarity = np.dot(baseline_vec, current_vec) / (np.linalg.norm(baseline_vec) * np.linalg.norm(current_vec))
```
This isn't about version upgrades, as the model name remains unchanged. The drift introduces non-negligible variance for applications relying on deterministic similarity searches or semantic hashing. Has anyone else performed longitudinal comparisons and identified mitigation strategies? I'm particularly interested if this is a known behavior with planned versioning, or if we should implement embedding version pinning at the API level.
—J
—J
That's a significant drop, and a cosine similarity shift from 0.998 to 0.972 on identical inputs is a major red flag for any production pipeline relying on vector stability. You've ruled out model version changes, but have you audited everything upstream of the actual inference?
You need to check the API's preprocessing pipeline. This includes:
* Text normalization (Unicode, whitespace, casing)
* Tokenizer behavior or vocabulary updates
* Any silent addition of system prompts or instruction templates
I've seen this before with other embedding services. The model card stays static, but the serving infrastructure around it evolves. Capture the exact raw HTTP request and response from both time periods if you can, and compare the tokenized input the model *actually* receives. That 0.026 delta is almost certainly a systemic change, not random noise.
—davidr
Cosine similarity dropping from 0.998 to 0.972 on identical inputs isn't a red flag, it's a predictable cost. You're paying for a black box service, not a deterministic function. That 0.972 is still extremely high for most similarity search use cases, and chasing that last 0.026 is where teams blow budgets on unnecessary recomputation and storage.
The real mitigation is to bake an expected variance budget into your SLA and stop assuming cloud APIs are static. If your application breaks from this level of drift, your architecture is too brittle, not their model.
pay for what you use, not what you reserve