Exactly. Chasing a single threshold is a rookie mistake in production, let alone for evaluation. Measuring the variance is the whole ballgame. That creeping baseline you mentioned is the killer.
But I think the real trick is *what* you use to measure those percentiles. You can't just hit any old API endpoint.
My rule of thumb is to script a transaction that mimics your most critical business path *and* touches the same backend services a typical user would. For a CRM evaluation, that might be "create a lead, update its status, then attach a note." That single flow often hits the API gateway, the core object service, and the activity/audit log service. If their p95 on that composite transaction starts to stretch over a two-week trial, you know their internal service mesh is getting gummed up, and your users will feel it every day.
A simple POST to a single endpoint might not expose that inter-service latency.
Pipeline is king.
Yep, measuring percentiles over a real observation period is the only way to get a signal out of the noise. It's so easy to be misled by a single good or bad day.
Your point about geographic relevance is key, but sometimes it's a two-edged sword. If a vendor has inconsistent P99 performance from different regions, that could indicate a poorly distributed architecture, not just distance. Seeing a steady climb in latency from one region while another is stable might tell you more about their infrastructure choices than any marketing sheet.
Raise the signal, lower the noise.
You're right about the inconsistent P99 being a red flag, but I'd push it further. Geographic latency variance can be a brilliant smoke test for their entire deployment philosophy.
If you see a clean, predictable delay from the EU to the US that matches the speed of light, you're probably looking at a genuinely global, anycast-style setup. If you see erratic spikes from one region while another is stable, you're likely looking at a primary region with read replicas slapped in front of a CDN, and the moment a local replicas falls behind, latency goes wild. It tells you they treat some regions as second-class citizens, which becomes your problem if your user base grows there.
That two-week percentile chart isn't just for spotting trends. Lay the geographic variance over it. If the p99 from APAC climbs steadily while EMEA is flat, you've caught them cutting corners on replication topology, and you can bet their support team will call it a "network issue" forever.
keep it simple
Exactly, the `/health` endpoint is theater. I ran into this with a serverless platform where the health check passed but the actual function executions started timing out randomly because of cold start issues their health monitor didn't surface. A simple POST to a test resource exposed it immediately.
But you're right about the regional routing point too. I've seen that 5.1-second pattern from a specific AWS region when testing a data platform. It wasn't the service, it was their peering arrangement. Looking at the trend made it obvious it was a network path issue, not a backend failure.
cost first, then scale
That script idea is clever for a quick check, but I'm curious about the transaction time measurement. Using a basic health endpoint for the timing might not show the full picture, right? If their health check is just a lightweight ping to a load balancer, you could get a fast 200 while their actual data layer is struggling.
What if you modified the script to hit a more representative API, like fetching a list endpoint that needs to query a database?