Hey everyone! 👋 I’ve been tasked with helping our team cut down our observability spend, and one of the big proposals on the table is to replace about half of our custom metrics with logs instead.
From what I understand, we have a ton of custom metrics (like counts or timers for specific user actions) that are driving up costs because of the high cardinality and the sheer volume of data points. Our engineering lead suggested that for a lot of these, we could just emit structured logs and then query them when needed, instead of tracking them as real-time metrics.
This sounds logical in theory, but I’m pretty new to the observability side of things. Has anyone here actually done something like this? I’m curious about the practical side:
* How do you decide which custom metrics are okay to move to logs? Is there a rule of thumb?
* Did you run into issues with alerting or dashboards that depended on those metrics?
* And maybe most importantly, what kind of cost savings did you actually see after making the switch?
We use a mix of tools (think Datadog for metrics, Elastic for logs) and I’m a bit worried about making our log volume explode and just shifting the cost there instead.
Any stories or lessons from your own cost-cutting projects would be super helpful! Thx!
You've correctly identified the main risk, which is a log cost explosion that negates any metric savings. Your vendor mix - Datadog for metrics, Elastic for logs - actually creates a perfect opportunity for a cost-benefit analysis. You need to compare the unit economics of an active metric series against the cost of storing and querying equivalent log data in Elasticsearch.
The rule of thumb for which metrics to move is temporal and criticality-based. Any metric used solely for retrospective analysis, not real-time alerting, is a candidate. User action counts for weekly reporting are ideal. However, metrics feeding automated alerts with sub-5 minute SLAs are poor candidates, as log-based alerting often introduces unacceptable latency.
Start by sampling. Don't move half at once. Pick the 5-10 most expensive, low-criticality metric series and replace them for a week. Measure the metric cost reduction and the exact log ingest increase. Use that data point to model the true breakeven point before committing to the 50% target.
I mostly agree, but the "unit economics" comparison is harder than it sounds. You can't just compare the per-series metric cost to the log ingest cost. You have to factor in the developer time spent refactoring code and the ongoing operational burden of building and maintaining dashboards from log queries. That labor cost often kills the ROI.
Also, don't forget to model query costs. You might save on ingest by storing logs, but if you're running frequent, complex aggregations in Elasticsearch to replicate a metric, those compute cycles aren't free. That's where these projects usually stumble.
Hey, great topic! We went through a similar exercise last year. That exact line about "making our log volume explode" kept me up at night 😅
We found the sweet spot was metrics for *internal* analytics, not system health. Think "user clicked export button" for a weekly product report, not "database query latency". The moment you need a real-time alert, logs are usually the wrong tool - the pipeline latency alone will drive you nuts.
One practical tip that saved us: before you refactor a single line of code, try to recreate your most important metric dashboards using *only* your existing logs for a week. Use Elastic queries to mimic the counts and timers. It'll expose the query cost and performance hit immediately, and you'll see if your team can even tolerate the dashboard load times. That experiment alone convinced us to keep about 30% of the metrics we initially targeted.
Good luck! It's a balancing act, but can be worth it.
null
You've hit on the key tension: swapping a known, predictable metric cost for a variable and potentially explosive log cost. The analysis has to be deeper than just a per-series comparison.
Start by categorizing your metrics by their *time-to-need* and their *dimensionality*. A metric like `user_action_count{action="export", region="us-east"}` used for a weekly internal report is a prime candidate. You can replace it with a structured log line emitted at the same point, and aggregate it on-demand in Elastic. However, if that metric has high cardinality across 20 different action types and 50 regions, your log aggregation query costs will multiply every time you run that report.
The biggest pitfall isn't the initial refactoring, it's the dashboard performance degradation. A metric is a pre-computed aggregate. A log-based dashboard must perform that aggregation at query time, which strains your Elastic cluster and slows down your dashboard's load time. You should prototype a few key visualizations using your existing logs first to measure this impact.
Ah, the classic "metrics to logs" shift! You've already hit on the main fear - cost shifting from Datadog to Elastic. Everyone's talking about the theory, but I've got a practical, code-first angle you should consider.
Instead of starting with a taxonomy of your metrics, start by instrumenting your logging library. The trick is to write a small wrapper that emits a structured log *and* increments a metric for a limited time. We did this for two months. The logs went to Elastic, the metrics still went to Datadog. Then we just turned off the metric emission in the wrapper and watched what broke. It caught a few sneaky alert dependencies we'd missed in our spreadsheets.
Your question about cost savings? They were real, but only for the right metrics. We saw about a 30% reduction in our Datadog bill for the migrated series. But - and this is key - our Elastic bill went up about 10%. The net gain was positive, but it wasn't the 50% some folks promise. The win was more about taming cardinality on the metric side than pure dollar savings.
> Did you run into issues with alerting or dashboards
Absolutely. Dashboards built on metric queries are snappy. Their log-based replacements were... not. We had to rebuild several Grafana dashboards to use heavier caching and pre-aggregated Elastic views. For alerting, we moved exactly zero critical, sub-5-minute SLO alerts. The pipeline latency for log-based alerts was a total non-starter for anything requiring real-time response.
pipeline all the things
That's a brilliant approach - the parallel run with the logging wrapper is such a smart way to de-risk the migration. It gives you concrete data on what actually breaks instead of theoretical models.
> Dashboard load times went from sub-second to 5-10 seconds
Exactly. That's the hidden tax. You saved on metric cardinality, but now your engineers are sitting there waiting for dashboards to load, which is a real productivity hit. We found that teams would stop checking certain dashboards altogether once performance degraded, which defeated the observability purpose.
Your net gain of +20% (30% saved minus 10% added) aligns with what I've seen. The real benefit isn't just the bill, it's simplifying your metric taxonomy. Did you find that removing those high-cardinality series made your remaining Datadog metrics more reliable? We saw fewer aggregation timeouts once we stopped hammering the system with millions of low-value series.
Prod is the only environment that matters.
You've hit on the core architectural question: what's the operational difference between a metric and a log? It's not just about cost, it's about the data's purpose in the pipeline. A metric is a pre-computed aggregate; a log is a raw event. Moving from the former to the latter trades off predictable query performance for storage flexibility.
The most effective criterion we used was based on data freshness requirements. If the business need is "within the hour," logs can work. If it's "within five minutes," you're building a real-time aggregation pipeline on top of your logging system, which often replicates the complexity you were trying to avoid. For your user action timers, map them against their alerting SLAs. Any metric tied to a PagerDuty policy should stay a metric.
On cost, we observed diminishing returns after the initial 20-25% migration. The savings from moving high-cardinality, low-urgency metrics were real, but the long-tail of dashboard queries against Elastic created a new, variable expense. The net effect was a modest reduction in the metrics line item, but a more unpredictable log analysis bill. The true win was a cleaner, more intentional metric taxonomy, which improved alert signal-to-noise.
—BJ
The "cleaner, more intentional metric taxonomy" is the only real win you mentioned, and I think it's the whole point. That predictability in your metrics bill? That's a feature, not a bug. Swapping it for a variable log analysis bill just trades a known problem for a mystery box.
You're right about the pipeline complexity, but it's worse. Once people start building those ad-hoc log aggregation queries for "fast enough" needs, you'll inevitably end up re-inventing a worse, less reliable metrics system inside Elastic. The complexity doesn't disappear, it just gets buried in dashboards no one can troubleshoot.
null
You're absolutely right about the hidden complexity. When we attempted a similar migration, we discovered teams weren't just running ad-hoc queries. They were building scheduled Kibana queries, saving the results as JSON, and then ingesting *that* back into a dashboard. This effectively created an undocumented, unmaintained ETL pipeline that became a single point of failure for several reports.
The "mystery box" variable cost became a secondary concern compared to the operational debt of these shadow pipelines. The predictability of a metric isn't just about budgeting, it's about a clear contract on data provenance and availability.
—BJ
Oh wow, the wrapper trick user200 mentioned is really clever - I'm gonna try that on our pipelines.
I'm also new to this, but we saw a similar thing with dashboard latency. We moved some low-priority ETL job status metrics to logs. The queries for our daily summary dashboard went from 200ms to like 8 seconds. The devs hated it.
Did you guys set up any aggregations or rollups in Elastic beforehand to help with that? Or is that just the trade-off you have to accept?
Exactly. That "clear contract" is why we keep core health checks as metrics. You can't have a log aggregation pipeline be your source of truth for whether a service is down. When teams start building those shadow ETL pipelines, they've already lost. You're just shifting the cost from your observability bill to the engineering hours spent maintaining brittle, undocumented junk.
show me the logs
You've nailed the core issue with those shadow pipelines. It's not just undocumented junk, it's *unmonitored* junk. When that ad-hoc Kibana aggregation fails silently because of a mapping change, nobody knows until a report is blank.
We treat core health metrics as a utility bill - predictable and non-negotiable. The cost isn't just in the line item, it's the guarantee of uptime and the team's time not spent debugging their own makeshift systems. Moving those is a false economy.
Data doesn't lie, but dashboards sometimes do.
> It's *unmonitored* junk.
That's the part most cost-cutting projects conveniently ignore. You can track your Datadog bill down to the penny, but how do you budget for three senior engineers spending a week untangling a failed log aggregation that was someone's "quick fix"? That operational debt always comes due with interest.
Treating core metrics like a utility bill is smart, but I think you're underselling the risk. The real false economy is when leadership sees the initial 30% savings on the observability line item and calls it a win, while the actual cost just gets buried in bloated sprint cycles and missed project timelines. No CFO is connecting those dots.
Show me the TCO.
You're absolutely right about the operational debt compounding with interest. The CFO sees the lower vendor invoice, but the true cost manifests as decreased feature velocity. Teams stop shipping because they're constantly firefighting data quality issues in those shadow pipelines.
This creates a second-order effect where engineering starts to distrust observability data altogether. When a dashboard is slow or a log aggregation breaks, they'll bypass the official tools and build yet another one-off system, further fragmenting your telemetry. You end up with multiple sources of truth, none of which are reliable.
The metrics bill isn't an expense, it's a reliability retainer. Cutting it directly reduces your ability to understand system state.
Boring is beautiful