We've been running a mid-sized Kubernetes deployment for about a year, using Prometheus for metrics collection and remote writing to AWS Managed Service for Prometheus. While the centralization has been beneficial for our teams, our monthly bill has consistently been 40-50% higher than our initial projections based on ingestion volume alone.
After a detailed cost analysis, I've isolated the primary cost driver to the remote write operation itself, not just the storage. Our setup is fairly standard:
* We're scraping approximately 150 targets with a 15s interval.
* We use a standard `remote_write` configuration with minimal label manipulation.
* Estimated active series hover around 200,000.
The discrepancy seems to lie in the interaction between our data model and the managed service's pricing structure. Key factors I've identified:
* **Metric Cardinality:** We suspect high cardinality, particularly from pod and container labels in Kubernetes, is inflating the number of unique time series being transmitted.
* **Write Amplification:** The default batching and retry logic in the Prometheus remote write client may be causing re-transmission of data during any brief ingestion hiccups, leading to duplicate charges.
* **Sample Volume vs. Cost:** The pricing model charges per sample ingested. At a 15s scrape interval, we are generating a significant number of samples per hour, and any inefficiency in the write path is compounded.
Has anyone conducted a thorough TCO breakdown comparing this remote write model to alternative approaches, such as:
* Running self-managed Cortex or Thanos for long-term storage, accepting the operational overhead.
* Using a different vendor's ingest model that charges primarily on compressed storage volume rather than per-sample ingestion.
* Aggregating or dropping high-cardinality metrics at the Prometheus level before remote write?
I'm particularly interested in concrete numbers on ingestion cost per million samples, and any configuration tweaks (like adjusting `batch_send_deadline` or `max_shards`) that had a measurable impact on your AWS bill without compromising data fidelity.
independent eye
Spot on about the cardinality from pod labels. Been there. That 15s scrape interval is also hitting you - it creates four times the write volume of a 30s interval before you even factor in any retries.
Two things that helped us rein it in:
* Aggressive `metric_relabel_configs` to drop high-churn labels like `pod_name` and `container_id` on the scrape side, before they ever get to remote write.
* Tuning `remote_write` queue config for our network - lowering `max_samples_per_send` and increasing `capacity` reduced those costly retries during ingestion blips.
Have you looked at your actual series count inside the managed service vs what your Prometheus server reports as "active"? That gap can be telling.
Data doesn't lie, but dashboards sometimes do.