Hey everyone,
I wanted to share a journey we've been on for the last quarter that's had a surprisingly big impact on our observability bill. Like many of you, we saw our costs for custom metrics start to spiral as our microservices ecosystem grew. Every new feature seemed to come with a new `business_*` or `process_*` metric, and our cloud provider's meter was just ticking away. We were spending thousands monthly just on those custom time-series.
So, we embarked on a project with a simple, but daunting, goal: **replace 50% of our custom metrics with structured logs**, and funnel those logs through a more cost-effective pipeline. The rationale was that many of our custom metrics weren't being used for real-time alerting or dashboards that refreshed every second. They were used for historical analysis, debugging, and weekly business reports—use cases where a slight delay (like a minute or two) was completely acceptable.
Here’s how we approached it, and what we learned:
**First, we did a brutal audit of our existing custom metrics.**
We categorized every single one into a simple matrix:
* **Need Alerting?** (Yes/No)
* **Required for Real-Time Dashboard?** (Yes/No)
* **Used for Historical Trends/Reporting?** (Yes/No)
* **Cardinality Level** (Low/Medium/High)
Anything that was a "No" for alerting and real-time dashboards, but a "Yes" for historical trends, became a prime candidate for migration. These were often things like:
* "Number of times a specific background job ran per customer type"
* "Status counts for non-critical workflows"
* "Aggregate feature adoption counters"
**The Migration Process:**
Instead of emitting a `Count()` or `Gauge()` via the metrics SDK, we now emit a structured JSON log with a clear `type` field (e.g., `"type": "business_metric"`). We include all the relevant dimensions as key-value pairs right in the log body. For example:
```
{
"timestamp": "...",
"level": "INFO",
"type": "business_metric",
"name": "document_export_completed",
"export_format": "pdf",
"user_tier": "premium",
"processing_time_ms": 1245
}
```
**The Cost-Saving Plumbing:**
This is where the magic happened. We route all logs with `"type": "business_metric"` away from our primary, expensive, real-time observability platform. We send them directly to a cheaper object storage (like S3) via a simple log shipper. Then, we use our existing data lake infrastructure (in our case, a Spark pipeline) to parse these logs nightly (or hourly, depending on need) and aggregate them into the formats our business intelligence tools expect. The cost difference is staggering—paying for cloud storage and batch compute is orders of magnitude cheaper than paying for high-resolution, queryable metric time-series data retained for months.
**The Trade-offs & Lessons:**
* **Latency is the biggest compromise.** Our finance team can't see up-to-the-minute counts anymore, but they were only looking at the data daily anyway. For engineering, debugging something that happened 5 minutes ago might require checking the raw log stream, which is still possible but not as instant as a metric query.
* **We had to tighten up our log schemas.** Moving business logic to logs forced us to be more disciplined about the structure and taxonomy of our log events, which was a positive side effect.
* **Not for everything!** Core application health metrics, latency percentiles, error rates—all that stayed as real metrics. This was purely for the "nice-to-know" business and auxiliary operational data.
For us, this project cut our custom metrics volume by over 60%, and the corresponding cost reduction from our observability vendor was even more pronounced. It required some cross-team buy-in, especially from data analytics, but showing them the potential savings got everyone on board.
Has anyone else tried a similar strategy? I'm curious if you ran into different challenges or found even more clever ways to slice the data. The battle against observability spend is real!
null