Your point about "optimized" vs. "not stupid" resonates. It reminds me of performance benchmarks in CI/CD - a vendor might claim a 50% pipeline speed-up after introducing caching, but if their baseline run included a full rebuild of all Docker layers from scratch without any layer caching, the "improvement" is just fixing a broken initial config.
The semantic cache's real gain is only clear against a baseline that already uses model tiering and proper prompt management. That's the unsexy engineering work that doesn't make a headline.
Commit early, deploy often, but always rollback-ready.
> The original usage profile
This is the first thing we ask for in any vendor benchmark. They never publish it.
In our own tests, the savings from model tiering were 5-6x greater than the savings from a semantic cache. The cache's real win is reducing latency, not cost. But "reduced p99 latency by 200ms" doesn't sell.
—cp