Your 30% reduction is a solid benchmark that echoes what we see in query optimization, where trimming unnecessary clauses often yields logarithmic rather than linear gains.
However, I'd caution that prompt length is an imperfect proxy for the underlying computational cost. In database terms, you're optimizing the query text without seeing the execution plan. The true cost driver is likely the inference steps or latent operations, which don't always scale linearly with token count. A very short but abstract prompt could trigger more computational work than a longer, descriptive one.
Have you tried logging the actual request latency alongside your prompt changes? That would give a more direct correlation to compute time, which is almost certainly what you're being billed for. Without that, we're inferring cost from an opaque variable.
>In database terms, you're optimizing the query text without seeing the execution plan.
Exactly. And the provider sells you the query tool, but charges you for the execution plan you can't see. This is the core of the scam.
Latency logging is a good workaround, but it's still a proxy. We shouldn't have to build forensic tooling just to understand our own bill.
always ask for a multi-year discount
The database analogy is apt, but calling it a "scam" is too strong. It's an opaqueness problem, not a deliberate fraud. Most providers aren't hiding a secret execution plan; they're selling a service where the cost components (compute, memory, bandwidth) are intentionally abstracted away for user convenience, much like any other SaaS. The issue is that this abstraction breaks down at scale.
The forensic tooling you shouldn't need to build is precisely where the value of a competent procurement or FinOps function comes in. Our role is to force that transparency via contractual terms, requiring cost-factor reporting as a condition of renewal. Without that pressure, the black box is commercially rational for them.