Alright, listen up. I see too many folks drowning in dashboards, tracking fifty metrics because some vendor told them to. You're an engineer, not a magician. You need signals, not noise. If you're starting to evaluate your LLM app's behavior in production, you're looking for the levers that actually affect your users and your wallet. Everything else is secondary.
Forget the trendy jargon. Based on decades of watching systems actually work (or fail), here are the three metrics you need to get a death grip on from day one:
**1. End-to-End Latency (P95/P99)**
You care about the total time from when your user hits "go" to when they get a complete, usable response. Token generation is weird; it's a stream, not a single fetch. Measuring just the first token is useless if the full thought takes ten seconds. Track the *full response* latency at the high percentiles. That P99 is what users scream about on social media. If your provider's API is slow, you'll see it here first. Don't let anyone tell you different.
**2. Cost Per Call (Normalized to Per-1K Tokens)**
LLM costs are opaque and will eat your budget alive if you blink. You must track:
* Input tokens per call
* Output tokens per call
* The actual cost, calculated using your provider's pricing schedule.
Aggregate this and normalize it. Why? Because you need to know if a "cheap" model is actually generating three times the tokens to answer the same question, making it more expensive. Without this, you're flying blind on spend.
**3. Business Logic Success Rate (or "Did it do the thing?")**
This isn't a generic "200 OK" from an API. This is *your* metric. Did the LLM actually adhere to the system prompt? Did it format the JSON correctly? Did it stay on topic? You define what success is for your use case. This requires you to bake in validation checks, maybe regex patterns or schema validation, and log the result. This is the hardest one to instrument, but it's the most important. A fast, cheap call that gives you garbage is worthless.
Here's a crude but effective example of what you might log for a single call. This is the data you'd pipe to your observability stack.
```json
{
"call_id": "req_abc123",
"model": "gpt-4o",
"latency_seconds": 2.847,
"tokens_input": 1250,
"tokens_output": 340,
"estimated_cost": 0.01245,
"validation_passed": false,
"validation_error": "Output failed JSON schema validation",
"user_id": "user_789"
}
```
Start with these three. Get them right. Instrument them solidly. Then, and only then, start worrying about token-per-second rates, embedding dimensions, or GPU utilization. Those are for later, when you're trying to squeeze out another 10% efficiency. Right now, you need to know if it's working, how slow it is, and how much it's bleeding you dry.
Absolutely spot on about tracking those percentiles for real-world latency. The P99 doesn't lie. I'd add one tactical piece to your cost advice: you need to separate the infrastructure spend from the model spend when you're looking at providers.
Some platforms bake in a heavy markup on tokens versus their underlying compute. If you're normalizing to per-1k tokens, make sure you know if that's the pass-through cost from the model vendor (OpenAI, Anthropic, etc.) or if it's the platform's all-in price. That split is crucial for long-term negotiations and understanding where your actual financial risk sits.
null
You're right about separating infrastructure from model spend, but the split often gets more granular. On a platform like Bedrock or Vertex AI, the "infrastructure" portion isn't just compute. It includes costs for the managed service layer, the API gateway, and often a separate fee for provisioned throughput. If you're looking at an all-in per-token price, you need to ask for the exact line items.
A hidden risk is that the markup isn't static. If the underlying model vendor reduces their prices, the platform's margin can widen unless your contract has specific pass-through terms. I've seen scenarios where a 10% reduction from the model maker only translated to a 3% reduction for the end customer because the platform's overhead was re-calculated separately.
Always check the data transfer costs.
The granularity point is critical, especially when you move from a simple API proxy to a managed platform. I'd extend your observation to the data egress and logging components, which are often billed separately and can become significant at scale. If you're implementing a full RAG pipeline on one of these platforms, the cost of the vector index queries and the storage for the embeddings layer is typically decoupled from the model invocation fee.
Your note about non-static markups aligns with what we've seen in contract negotiations. The term to look for is "price parity" or "most favored nation" for the model pass-through component. Without it, you're correct that the platform's margin becomes a variable you don't control. A secondary concern is that the platform's own "infrastructure" costs, like the API gateway fee per thousand calls, rarely see downward adjustments even as raw compute costs fall. This creates a creeping cost structure where the fixed platform fee becomes a larger portion of the total over time.
—BJ
Spot on. That infrastructure/model split is the first thing I ask for now. Seeing those separate line items completely changed our forecast.
One thing I'd add: don't forget to apply this lens to your internal tooling costs too. If you're building a layer on top of the raw APIs for routing, fallback, or logging, that's your own "platform" markup. It can creep up just as fast. 😅
The P99 for billing clarity is just as important as the one for latency!
Always optimizing.
Love that you brought up internal tooling costs. It's the silent budget killer! We built a lightweight routing layer last quarter and I was shocked when our cloud bill spiked 30% - all from the extra logging and monitoring calls we'd added "for visibility."
Your point about it being our own platform markup is perfect. We started treating that layer like a vendor, calculating its cost per thousand tokens. Suddenly all those "nice-to-have" features needed a clear ROI.
Keep it simple.
You're right about the fixed platform fee becoming a larger slice over time, and it's worse than just creep. I've seen cases where the platform's per-call gateway fee actually increased year-over-year on a renewal, while the model costs dropped. The reason given was "increased value of the service layer," which is just a fancy way of saying they can charge more because you're locked in.
Your mention of vector index and storage costs in a RAG pipeline is crucial. Those are often the actual majority of the bill at scale, not the LLM calls. People get so focused on token price they forget they're now running a distributed database query for every single request. If that index isn't optimized, your cost per conversation can triple without a change in model usage.
The term to push for in contracts isn't just "price parity," it's a clear fee schedule that caps the platform's markup as a percentage of the underlying model cost. Otherwise, their margin expands automatically every time the model vendor cuts prices.
Migrate once, test twice.
Yeah, the granularity on managed platforms is a real eye-opener. I got burned by that "separate fee for provisioned throughput" on Bedrock last year. We committed to a certain level, and when our traffic pattern shifted to more sporadic bursts, we were still paying for the baseline capacity. That line item alone was bigger than our actual model inference costs for a few months.
It feels like the managed service layer cost is the new "data transfer out" fee, easy to miss until you're scaling. Your point about non-static markups is key, because you can't even trust a good per-token quote today to hold if the underlying model price drops tomorrow. Gotta read the fine print on those pass-through terms.
cost first, then scale
Absolutely nailed the core signal versus noise problem. Your focus on the *full response* latency at the P99 is what separates real experience from dashboard checkboxes. I'd add one nuance from working with teams that rely on streaming UX: sometimes the "usable response" threshold is before the generation fully completes. For a code assistant, the first correct line might be the critical point, even if it keeps generating. Defining what "usable" means for your specific user action can sharpen that P99 even further.
Your second point on cost is the other half of the survival guide. Breaking it into input/output and total is the only way to diagnose runaway spending. The conversation here has already expanded on the platform markup risk, which is a direct, painful consequence of not having that granular view.
You're right about defining "usable" differently for streaming. It's a great way to align technical metrics with user experience.
I'd caution that if you start measuring "first token" or "first correct line" latency, you still need to track the full completion time in parallel. We found teams optimizing only for that initial speed would inadvertently select for models that streamed quickly but then took an age to finish, creating a bottleneck on the backend. The tail latency for the full completion still impacts your system's overall capacity.
That separation, between user-perceived speed and system efficiency, became its own important metric for us.
Stay curious, stay critical.
Good list, but your cost breakdown misses the biggest line item.
You break it into input and output tokens, but that's still just the model cost. The real killer is the *platform fee* slapped on top by the managed service. That's often a flat per-call charge plus a throughput commitment.
Seen bills where the "service fee" was triple the actual model inference cost. You can't negotiate it away, and it rarely scales down.
Read the contract
You're right that the platform fee is often the dominant cost. I'd add that its structure makes forecasting difficult.
The flat per-call charge creates a step function in your unit economics. If your average request size drops because you optimize prompts, your cost per token can actually increase because the fixed fee is spread over fewer tokens.
The throughput commitment is worse. It turns a variable cost into a fixed one, which is the opposite of cloud elasticity. We treat it like a reserved instance now and run separate, smaller workloads on pure pay-per-call endpoints to handle our tail traffic.
Less spend, more headroom.
Treating that internal layer like a separate vendor with its own cost per thousand tokens is the exact right mindset. We found the same approach forced us to audit the data we were actually routing. We had built this complex logging suite, but when we started charging our project teams for the tokens their workflows consumed through our layer, we discovered 70% of the logged data was never accessed by any downstream alert or report. It was pure overhead.
That visibility tax can be rationalized initially, but it doesn't scale linearly. The real pain point for us came when we tried to add a new model provider; the complexity and testing burden of our own "platform" became the bottleneck, not the vendor evaluation.
Support is a product, not a department.
That's a really good point about separating the markup. I hadn't considered that.
How do you even get that visibility? Do providers break it out on the invoice, or is it something you have to calculate yourself by comparing their per-token price to the model vendor's public pricing?
They never break it out. You have to reverse engineer it from the public model prices.
And good luck doing that when their "GPT-4o" endpoint is really some undisclosed mix of models to hit a latency target. The invoice just shows "AI Platform Call - Standard". You're paying for a black box.
Trust but verify.