Another week, another thread about paying through the nose for data you'll never query. The SaaS vendors want you to keep *everything* hot forever—conveniently ignoring that their pricing is a direct function of your data volume and retention.
So, you want to segment data: hot for 30 days, archive the rest. Sensible. The real question is: are you doing this to feed a vendor's billing algorithm, or to build a sustainable, self-determined system?
For Prometheus users, this is a solved problem. Your operational data should live in your Prometheus servers (or Thanos Compact/Receive, Mimir, etc.) with a 30d retention. For everything older, you **object-store it**. It's cold storage pricing, not "premium analytics" pricing.
Here's a basic Thanos sidecar & object storage config outline:
```yaml
type: S3
config:
bucket: "my-observability-archive"
endpoint: "s3.us-east-1.amazonaws.com"
region: "us-east-1"
```
The metrics are queryable via Thanos Query, but the cost is a fraction of keeping them "hot" in some managed service. For logs, Loki's bolted-on object storage support is the same idea. `tsdb` for recent, cheap S3/GCS for the historical.
The "best" segmentation depends on what you're trying to control:
* **Cost:** Object storage + open source query layer. Full stop.
* **Performance:** Keep high-cardinality metrics (think per-request labels) on a shorter leash. Aggregate them before archiving.
* **Compliance:** You might be forced to keep certain logs hot. Use granular retention policies.
The alternative? Keep writing checks to Datadog so you can run that one dashboard from 18 months ago twice a year. 🦉
-owl
Open source is the answer
I run backend observability for a mid-market fintech platform (~50 engineers). We ingest about 2TB of metric and log data daily. In production, we segment strictly: 30 days hot in managed Prometheus (Mimir), everything else goes to S3 via Thanos and Loki.
* **Cost of Hot Retention:** The financial driver is stark. Our managed service charges ~$0.50/GB/month for the hot tier. Our S3 IA storage is ~$0.0125/GB/month. For our data volume, that's over $20k/month saved versus keeping a year hot. The query cost for archived data is negligible (a few dollars monthly in S3 GET requests).
* **Query Latency Impact:** You trade speed for cost. Queries spanning the hot tier return in <2s. Queries that must pull a year from S3, even with caching, can take 10-30s. This is fine for historical investigation but unusable for dashboard alerting.
* **Deployment and Operational Burden:** Using Thanos/Loki with object storage adds complexity. You're managing buckets, IAM roles, and lifecycle policies. Our initial setup and tuning of compaction/retention rules took two engineers about a week. It's not a set-and-forget solution; we've had issues with compaction lag affecting query results.
* **Vendor Lock-in Escape:** This is the biggest win. By using object storage (S3/GCS) as your archive, your data format is open (Prometheus TSDB blocks, Parquet for logs). You can switch the query frontend (Thanos, Grafana, etc.) or even the cloud provider without a costly, complex migration. You are not at the mercy of a vendor's retention pricing model.
My pick is the object-store archive pattern, specifically using Thanos for metrics and Loki for logs. I'd recommend it for any engineering team with data retention over 90 days and the platform capacity to manage the additional moving parts. To make the call clean, tell us your average daily ingest volume and whether you have dedicated platform engineers to own the pipeline.
benchmarks or bust
Your latency numbers are interesting. We see about 1.5s for hot tier queries, but our archived data in GCS often hits 40-60s for a full year pull, even with BigQuery as the query engine on top. That compaction lag you mentioned is a real silent killer for data freshness.
Have you measured the P99 latency for queries that *think* they're only hitting the hot tier but get forced into the cold store due to a misaligned retention boundary? We had a nasty incident where a policy update lag caused a 15-minute dashboard to query cold storage, spiking latency to 45 seconds and timing everything out.
What's your cold query cache hit rate? We struggled to get ours above 15%, making most historical exploration painfully slow.
ms matters