Everyone's obsessed with dashboards and distributed traces. But half the queries are slow because someone's SELECT * on a trillion spans. You can talk about cardinality limits all day, but the real test is: can you get a simple P95 latency *fast* when you need it?
Wrote this to cut through the marketing. It's not comprehensive, but it hits the basic "time to useful number" test. Replace the endpoints and tokens. Run it when you're not in a firefight—results vary wildly under load.
```python
import time
import requests
import statistics
platforms = {
"cool_observability": {"url": "https://api.cool.io/query", "token": "YOUR_TOKEN"},
"legacy_monitoring": {"url": "https://legacy.com/api/v1/query", "token": "YOUR_OTHER_TOKEN"}
}
query = 'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))'
latencies = {}
for name, config in platforms.items():
times = []
headers = {"Authorization": f"Bearer {config['token']}"}
for i in range(10):
start = time.time()
r = requests.post(config['url'], json={"query": query}, headers=headers)
r.raise_for_status()
elapsed = time.time() - start
times.append(elapsed)
time.sleep(0.5) # be polite
latencies[name] = {
"avg": statistics.mean(times),
"p95": statistics.quantiles(times, n=20)[18] # 95th percentile
}
print(latencies)
```
Ran this against a couple vendors. One "modern" platform averaged 2.3 seconds. The boring one did it in 0.8. Food for thought before you commit to a three-year contract based on a slick demo.
Keep it simple
Your script's timing loop is measuring local network hops and the vendor's API gateway. Not the actual query execution in their distributed system. The "time to useful number" test gets skewed by their edge caching and how they prioritize small programmatic requests.
Run that same test during a regional outage when their platform's under load. Or better yet, try it for a query that spans a week of data. That's when the shiny demos crack and you see what you're really paying for in that enterprise contract.
What's the P95 of the P95 query?
Your mileage will vary
Good point about network hops skewing it. That's why I run these tests from multiple cloud regions I actually operate in.
Your "P95 of the P95 query" comment is the real test. I log the variance across 100 runs, not just the mean. The platforms with consistent low variance win during incidents, even if their average is higher. The ones with a great average but a 10s tail are useless when you're on fire.
Also, the script needs a cold start vs warm cache test. Hit the endpoint after 10 minutes of silence, then immediately again. The delta tells you more about their architecture than the raw number.
Benchmark or bust
Agreed on variance over mean. I've seen platforms with a 1.2s average and a 15s P99. That's a hard no for runbooks.
Cold start test is good, but don't wait 10 minutes. Their cache TTL is often 5 minutes or less. Hit it after 15 minutes of silence, then three rapid repeats. The pattern shows their cache invalidation logic and if they have a fast path for identical queries.
Also add a memory test: run it 100 times, store all latencies in an array, don't calculate stats until the end. Some client-side stats libs skew the timing.
Trust, but verify
Your script measures API responsiveness, not query execution. You're testing their edge network, not their query engine.
That `histogram_quantile` is trivial for any warm backend. The real test is a query they can't cache - something with a new random label or a fresh time window. If you're not hitting their query planner with something unique, you're just benchmarking their CDN.
Also, you're doing 10 sequential runs. That warms up your local connection and their API tier. For a "time to useful number" test, you need a single cold request after a period of inactivity. That first latency spike is the one that matters at 3 a.m.
- Nina