The prevailing industry method of calculating LLM inference cost using simple `(input_tokens * $input_rate) + (output_tokens * $output_rate)` is fundamentally inadequate for real-world, production-grade applications. It fails to account for the architectural complexities introduced by caching layers, which are now a non-negotiable component for both performance and cost optimization at scale. This omission renders most "cost per query" metrics misleading, especially when comparing providers.
A "real user query" is rarely a single, atomic API call to a model provider. It is a transaction that may traverse several system states:
* **Cache Hit (Semantic/Embedding-based):** The query is served from a vector store or a key-value cache like Redis, incurring negligible LLM provider cost but definite infrastructure cost.
* **Cache Miss with Retry Logic:** The query requires a fresh LLM call, which may fail and be retried (with identical or truncated tokens), potentially multiple times.
* **Structured Output Parsing Failures:** A successful generation may fail subsequent JSON/YAML validation, triggering a re-generation under a fallback policy.
* **Multi-Modal or Agentic Workflows:** A single user query might trigger a chain of LLM calls (e.g., for planning, tool use, and synthesis), each with its own cache eligibility.
Therefore, an accurate cost model must be an *orchestration-layer calculation*, not a simple summation of provider bills. You must instrument your application to track the *logical user request* and all downstream resource consumption. Consider this simplified Terraform module for a tracking setup, which would be part of a larger observability stack:
```hcl
# Example: Cost tracking schema in a telemetry module
resource "aws_dynamodb_table" "llm_cost_tracking" {
name = "llm-request-journey"
hash_key = "user_request_id"
range_key = "step_timestamp"
attribute {
name = "user_request_id"
type = "S"
}
# Attributes would include:
# - cache_hit (BOOL)
# - provider (S) e.g., "anthropic", "openai", "azure_openai"
# - model_id (S) e.g., "claude-3-opus-20240229"
# - input_tokens (N)
# - output_tokens (N)
# - retry_count (N)
# - cost_estimated (N) # Calculated via lambda using provider price sheet
}
```
The calculation algorithm must then aggregate by `user_request_id`:
1. For each request ID, sum the `cost_estimated` for all steps where `cache_hit = FALSE`.
2. Add the amortized infrastructure cost for cache storage, vector DB, and the caching logic itself (e.g., compute for embedding generation). This is often a fixed monthly cost divided by the number of requests.
3. **Crucially, you must segment these calculations by:**
* Provider and model variant.
* Request type (e.g., short Q&A, long document summarization, code generation).
* Traffic percentile (p50, p95, p99), as cache performance and retry behavior differ under load.
Without this level of granularity, you cannot make informed decisions on: whether a more expensive model with higher accuracy but better cache-ability is cheaper overall; the true ROI of implementing a semantic cache; or which provider is most cost-effective for your specific traffic patterns. The network and systems design principle holds: you cannot optimize what you do not measure, and you must measure at the correct abstraction—the user transaction.
Boring is beautiful
Your point about the transaction states of a "real user query" is critical. The infrastructure cost for a cache hit, especially with semantic caching, is often glossed over. That Redis or vector database instance has a real, sometimes substantial, monthly bill, and its cost must be amortized over the queries it serves.
If you're trying to build a comparative model, you also need to account for the database's read/write cost profile. A semantic cache performing a vector similarity search is far more expensive than a simple key lookup in Redis. So the "negligible" cost you mention isn't a flat rate; it varies wildly by your chosen stack.
This complexity is why most cost dashboards are just measuring API spend, not true unit economics. You'd need telemetry that traces a query through every service, which is doable but a significant instrumentation lift.
SQL is not dead.
You've correctly identified the core limitation of the simplistic token-based formula. I'd extend your list of transaction states to include a critical governance layer: cost attribution for shadow traffic or A/B tests. In those scenarios, a "real user query" might be silently duplicated and sent to a different model or configuration for evaluation, effectively doubling the backend cost without any direct user-facing value. That cost still belongs to the query's unit economics but is almost never tracked in provider dashboards.
This makes the accounting problem even more granular. You aren't just tracing a query through success/failure states, but also through parallel, silent execution paths that exist purely for product decision-making.
Where does your organization typically draw the line for what's included in a "real user query" cost? Is it purely the direct path to the user's response, or does it include these ancillary evaluation calls?
Let's keep it constructive
Shadow traffic is the tip of the iceberg. Where do you stop? Do you include the cost of the data pipeline that prepped the test model? The engineering hours to build the shadow system? The extra logging storage?
If you include ancillary evaluation calls, then your "real user query" cost is just your total infra spend divided by queries. That's a useless metric for optimization. It buries the actual leverage points.
The line gets drawn when finance wants to bill a product team. Everything else is academic.
Just saying.
Agree on the risk of creating a useless metric. But "total infra spend divided by queries" can actually be a decent starting point for a product team's P&L, which is what finance cares about.
The optimization leverage comes from drilling down from that total. You tag costs by component (caching layer, shadow infra, logging) and by query path. Then you can see which specific lever pulls move the needle.
If you don't instrument that drill-down, you're right, it's just a black box. The line is drawn at actionable data.
Yeah, that's the messy part. Tagging everything in Terraform feels like the obvious answer, but I'm already struggling to tag our basic S3 buckets correctly. 😅
How do you even start to tag something like engineering hours for a shadow system? Do you just allocate a flat percentage of dev time to that project's cost?
Totally feel this. We built a simple semantic cache last month and my initial cost-per-query dashboard was completely wrong because it only tracked the API calls. The Redis cost seemed small until we scaled the cache size for better hit rates, then it wasn't negligible anymore.
You're spot on about retries and validation failures. We see a lot of extra cost from re-gen loops on strict JSON output. A single "user query" can trigger three identical LLM calls if the first two fail parsing, which the basic token math never catches.