Everyone talks about tokens and latency. Those are table stakes. If a vendor demo doesn't move past that, they're selling infrastructure, not a solution you can actually use.
You need to ask for metrics that map to real operations breaking down. Here’s what I push for:
* **Error rates by use case, not overall.** A 0.5% overall error rate is meaningless. What's the rate for "classification" vs. "summarization" vs. "data extraction"? If they can't segment it, they aren't monitoring properly.
* **Latency at the 95th and 99th percentiles for your specific tasks.** Average latency is a fantasy. You need to know the worst-case delays for a live chat summary or a lead scoring call. Ask them to simulate your exact payloads.
* **Degradation under sustained load.** Can they show performance over a 2-hour period mimicking your peak traffic? Output quality often tanks first—ask for consistency scores across the run.
* **Real cost per *operation*.** Cost-per-token is a math puzzle. Make them give you the estimated cost to process 1000 support tickets, or to enrich 1000 leads. That's the number that matters to finance.
* **Model update transparency.** What's their process and lead time for major model version updates? You don't want your parsing jobs to break on a Tuesday because they pushed an update.
Most demos show a perfect, isolated query. Your production environment is not that. Ask for the data from their worst client in the last quarter.
Show me the workflow.
Lisa M.
This is super helpful, thanks. The "real cost per operation" point really hits home. I was getting lost in token math, but framing it around 1000 support tickets makes it concrete for my team's budget.
What's a good way to actually ask for that data in a demo? Do I just say "show me a consistency score over 2 hours" or is there a better way to phrase it so they actually run the test?
Still learning.
Exactly. Don't ask for the test, ask for the dashboard. Say "Can you pull up the monitoring view you'd use to check that live?" That shows if they actually have it built in.
If it's just a static slide, that's a red flag for me.