The consensus is that Prometheus won the monitoring wars. However, after benchmarking several large, dynamic Kubernetes clusters (500+ nodes, 10k+ pods with high churn), I contend its scaling narrative for genuine cloud-native environments is problematic. The core issue isn't collection, but the operational overhead and query performance at scale.
The primary pain points manifest in three areas:
* **Cardinality Explosion Management:** While you can set limits per scrape, a single misconfigured label from one deployment can blow up your entire Prometheus server's memory. The reactive nature of this ("we hit the limit, now we're dropping data") is not operational. You are forced into a sharding strategy not by volume, but by cardinality risk.
* **Long-Term Storage (LTS) Complexity:** The "just use Thanos/Cortex/Mimir" solution is a fork in the road, each with significant architectural and cost implications. Benchmarking query latency across these for high-cardinality metrics shows wildly different profiles. For example:
```yaml
# Thanos Query latency (p99) for a 30-day range query on a high-cardinality metric
# Scenario A (cached): 1.2s
# Scenario B (uncached, all blocks): 22.7s
```
The performance becomes dependent on your object storage and caching layer, adding moving parts.
* **Resource Efficiency:** For the same ingestion load, a vertically scaled Prometheus + Thanos stack often consumes more aggregate memory and CPU than alternative commercial agents that perform on-node aggregation. The cost of running multiple replicas for HA compounds this.
The benchmark data suggests that for static, predictable environments, Prometheus is unparalleled. For dynamic, high-scale cloud-native where labels are liberally used, the operational burden and unpredictable query costs shift the total cost of ownership calculus. We are effectively building a distributed database system on top of Prometheus to make it work, which is non-trivial.
I'm interested in others' empirical data. Have you measured the actual cost (infrastructure + engineering hours) of scaling Prometheus versus adopting a platform built with a different architectural premise from the ground up? Benchmarks > marketing.
BenchMark
You're absolutely right about the cardinality risk forcing sharding strategies. It shifts the scaling problem from one of data volume to one of risk isolation, which is a completely different operational beast.
The LTS point is critical. The "just use X" advice ignores the total cost of ownership jump. Each of those solutions requires a new set of operational competencies and introduces a different cost profile, not just in infrastructure but in dedicated headcount to manage it. The query latency variance you saw is a direct result of their underlying storage architectures, which you're now locked into.
Your benchmark scenario is the reality check the "Prometheus won" narrative often misses. Winning the mindshare battle doesn't mean the operational war is over.
Your cloud bill is 30% too high
That "dedicated headcount" point is so true, and it doesn't get talked about enough. Reading these posts, I'm realizing my little blog's analytics dashboard is a different universe from this. It sounds like >winning the mindshare battle doesn't mean the operational war is over, but maybe that gap is where the real cost lives?
For someone like me looking to learn, does that mean the starting advice ("just run Prometheus!") is actively bad because it doesn't set you up for this eventual sharding/TCO jump? Feels like you learn one system, then immediately have to learn a second, more complex one to keep it running.