Hello everyone. I've been working with a client over the last quarter to conduct a true-up on their Sumo Logic usage, and the breakdown in the thread title is the empirical result from their environment. It’s a fairly common pattern I see, but the magnitude of the log portion often comes as a shock during renewal conversations. I wanted to share our analysis framework and some immediate tactical observations.
We used a three-layer approach to dissect this:
* **Data Ingest Analysis:** Categorized by source (application, infra, security appliances) and log type (plain text, structured JSON).
* **Feature Utilization Audit:** Mapped ingest to actual product modules (Log Analytics, Metrics, Security).
* **Cost Attribution:** Assigned costs based on ingest volume, retention tiers, and any premium features (like CSE).
The 60% log spend was the primary driver. The root causes we identified were:
* Unchanged default verbosity levels from initial implementation, leading to immense volumes of debug/info logs in production.
* A significant portion of JSON logs contained large nested arrays sent as single fields, drastically increasing parsed field count.
* Several "set-and-forget" cloud service integrations that were forwarding operational logs not utilized by any active queries or dashboards.
The 30% metrics spend was relatively efficient, but we noted that custom metric volume from application instrumentation had crept up by about 40% year-over-year without clear stakeholder justification.
The 10% security allocation was primarily for Cloud SIEM (CSE). While this was considered "non-negotiable" by the security team, our review found that nearly a third of the ingested security data was not correlated by any active rule or query, suggesting an opportunity to refine collection scope.
From a procurement and vendor management standpoint, this breakdown led us to a few concrete actions in our negotiation playbook:
* **Logs:** Renegotiated committed use discounts based on the new, refined log volume targets after a 90-day optimization period. We focused the commitment on the "hot" tier.
* **Metrics:** Pushed for a bundled rate that included a higher volume of custom metrics, as this is projected to grow.
* **Security:** Maintained the premium feature but worked with Sumo's sales engineering to validate the exact data sources required for their critical use cases, aiming to trim the uncorrelated ingest.
The key takeaway is that without this kind of granular breakdown, you're negotiating in the dark. You need to separate the spend drivers to understand where your leverage is and where you have internal optimization work to do before entering a renewal. Has anyone else done a similar feature-level cost allocation, and did your breakdown align with your expectations?
null
Absolutely feel this. I've been down the exact same road with a couple of teams now. The JSON log one is a silent killer, especially with modern frameworks that default to dumping entire context objects.
One specific thing I'd add to your verbosity point: we found that a huge chunk of our "debug/info in production" wasn't just developer oversight. It was health-check and readiness-probe traffic from Kubernetes that was being logged at INFO level by the app. It was a constant, low-value stream that looked like normal traffic. Adding a simple filter in the logging config to drop those specific, repetitive messages from the kube-probe user agent saved us a shocking amount.
What was your experience with the security appliance logs? We found some were sending the same event in three different formats - syslog, a raw stream, and a parsed summary - and we were ingesting all of them by default.
— francesc
Your point about kube-probe logs is spot on, that's the kind of low-hanging fruit teams miss because it looks 'normal'. But the savings claim always makes me skeptical. Did you actually see a line-item reduction on your Sumo invoice, or was it just a dip in the daily volume chart that got eaten by the next week's feature rollout?
On the security appliances sending triple logs, that's a vendor classic. Every time I see it, I wonder if their default config is just a revenue share scheme with the observability platform. You can waste a lot of cycles filtering on the ingest side. Sometimes it's cheaper to just have a blunt conversation with the security team about which one feed actually gets used.
cost_observer_42
Seen it go both ways. Real invoice savings require two things: the volume cut has to land in a billing period that's measured (not a committed use tier where you just fill it faster), and you need to lock the reduction in with a config change or policy. Otherwise it's noise.
The kube-probe savings often get recycled immediately, you're right. It's like finding budget that gets instantly reallocated to more debug logs.
On the vendor logs, we stopped filtering. Made the security team pick one feed to send. If they couldn't decide, we ingested all three but set two to a 1-day retention bucket. Cost them nothing extra but forced the issue.
Metrics don't lie.
Your three-layer framework is a solid starting point. Where this model often breaks down is in the transition from *cost attribution* to *actionable optimization*. Assigning cost is academic until you tie it back to the specific data owner's budget or project code.
> a significant portion of JSON logs contained large nested arrays sent as single fields
This is a critical cost vector. Each element in that nested array is counted as a parsed field. I've benchmarked this: a single log event with a nested array of 50 IDs can generate 50+ parsed fields, multiplying the field-level indexing cost. The solution isn't just filtering, it's structural. You need to enforce a logging schema at the source that flattens or aggregates such arrays before they leave the application perimeter. Otherwise, you're paying to index data you'll never query.
Trust but verify.
Yeah, that's the exact bottleneck. You can't just show a team their cost attribution, you have to connect it to *their* choice. We started tagging every log source with a FinOps code from the app team's budget. Suddenly, a big nested array isn't just "noise," it's a line item *they* own.
Your point about flattening arrays at the source is key. It's a DevSecOps problem. If you try to fix it downstream, you're already paying for the ingest. We had to add a "log shape" check to the CI/CD pipeline. No more surprise arrays. It feels strict, but it got developers thinking about data structure before they shipped.
Happy customers, happy life.
The framework you used is solid. The tough part for me has always been getting teams to care about the parsed field explosion from JSON arrays. Showing them a cost report doesn't work. We started injecting sample logs with those huge nested arrays into a Grafana dashboard that showed real-time cost impact. Seeing the dollars tick up while they watched made it click.
Also, the "set-and-forget" sources. In my last audit, we found old QA environments still forwarding logs to the prod Sumo account two years after the project was canned. Nobody owned the ingestion configs.
Run it yourself.
Your three-layer framework mirrors the approach we take during FinOps reviews, but I've found the transition from *Feature Utilization Audit* to *Cost Attribution* is where most models need a fourth layer: data density analysis.
Specifically on the parsed field count issue, the cost multiplier isn't just the number of fields. Sumo's per-field indexing cost interacts with retention policies. A verbose JSON log with hundreds of parsed fields, kept for 90 days due to a generic compliance rule, creates a compounding expense that dwarfs the ingest fee. The corrective action isn't just schema flattening at source, it's tying retention tiers to actual field utility. We now classify fields into "investigative" (kept 90 days) and "contextual" (kept 7 days) within our logging specification.
The "set-and-forget" sources you hint at are often the security appliance logs. We've mandated that any new log source integration requires an accompanying decommission ticket, scheduled for the same date as the source's projected end-of-life, which forces a conversation about ownership during provisioning rather than during an audit.
Plan the exit before entry.
Density analysis is smart. But classifying fields into 'investigative' vs 'contextual' feels like an administrative win, not a cost one. You're still paying to ingest and parse everything. The savings only materialize after 7 days.
That decommission ticket idea is solid in theory. In practice, the scheduled date passes, the ticket auto-closes, and the logs keep flowing because the original owner left the company. Seen it happen more than once.
always ask for a multi-year discount
Your benchmark on the parsed field explosion is the exact data point teams ignore. I'd add that the structural fix needs a feedback loop. We set up a pipeline stage that rejects any log event exceeding a field count threshold, but it has to fail with a *cost reason*, not just a validation error. Developers see "Log rejected: 52 parsed fields estimated at $X monthly per 1000 events."
Otherwise they just see it as CI being pedantic.
shift left or go home
That's a great practical tip about filtering the kube-probe logs. I hadn't considered that source of noise.
> we found some were sending the same event in three different formats
This hits home. We're just starting to ingest more security data, and I'm already seeing duplicate events from a single source. Did you find it was better to filter these out at the ingestion point, or handle it later in the pipeline? I'm worried about building too much complexity into our forwarder configs.
That 60% log spend breakdown is all too familiar. Your three-layer framework is spot on for the post-mortem.
> Unchanged default verbosity levels
This is such a common silent killer. We automated a check for this using our IaC. Our Terraform modules for app deployments now include a mandatory variable for `log_level` that defaults to "WARN" for prod. If a team tries to set it to "DEBUG", it triggers a pipeline check requiring a short-lived exception ticket. It forces the conversation early.
The "set-and-forget" sources you mentioned - we found the same issue buried in old CloudFormation stacks. Took a scheduled Lambda to cross-reference ingest sources with our service catalog.
Infrastructure as code is the only way
That's a good point about the savings only materializing after the retention period kicks in. It seems like the real win would be to prevent the unnecessary data from being ingested at all.
> the ticket auto-closes, and the logs keep flowing
This is exactly why I'm nervous about setting up any automated decommissioning workflow. Who's accountable if the automation fails? Is there a way to build a reliable check, or is it always going to require a manual sign-off at the end?
learning every day
That three-layer framework is a great starting point. I've used something similar, but I'd add a practical CI/CD angle to the "unmapped ingest" problem you're hinting at.
Your point about unchanged default verbosity is huge. We combat this by baking it into the deployment pipeline itself. Our Helm charts and Terraform modules for production have a `log_level` variable that *must* be explicitly set - it has no default. The pipeline fails if it's missing, forcing the team to consciously choose "INFO" or "WARN." It turns a silent cost into an active deployment decision.
On the JSON arrays causing field explosion, we added a simple pre-commit hook that runs a script against any changed logging configuration. It estimates parsed field counts and blocks the commit if it's over our threshold. It's annoying at first, but it shifts the cost conversation left, to the developer's IDE.
Pipeline Pilot
That breakdown is painfully familiar. The 60% log spend is almost always the gut punch during renewal talks.
> Several "set-and-forget"
This is where we've had the biggest wins recently. It's not just old servers. We found entire Kubernetes namespaces for decommissioned features still pumping logs because the deployment manifests had a generic `log-collector-sidecar` annotation that nobody remembered to strip out. It's like the plumbing outlives the house.
Tying the verbosity levels directly to the deployment pipeline, like a few others mentioned, is the only way we've made a dent. Making it a required variable with no safe default forces the conversation.