Your point about premium features like CSE in the cost attribution layer is spot on and often overlooked. Mapping ingest to a module isn't enough; you have to validate if the data is actually *processed* by that module's logic. I've seen teams routing firewall logs through a security module at triple the cost, but their only "detection" was a count-by-IP dashboard they checked twice a year.
That validation step is where the real savings hide. A quick script to compare sources tagged for CSE against active rule matches over the last 90 days usually reveals a few pipelines that are just coasting on the premium tier. You can often downgrade them to basic analytics with zero operational impact, but you need that audit to prove it.
That script validation is such a powerful idea. We've all seen the dashboards that get built once for a project review and then forgotten, but paying a premium for the data feeding them is a whole other level of waste.
We took it a step further and tied it to our vendor review cycle. During renewals, we require teams to show not just that a data source is tagged for a premium module, but also the last three high-severity alerts or automated actions it generated. If the answer is "none," we start the justification for moving it to a standard tier right there. It turns the audit from a nice-to-have into a contractual checkpoint.
Ask me about my RFP template
That's a smart escalation. Tying the validation to the renewal forces a hard look at value, not just legacy tagging.
It makes me wonder about the practical side, though. In your process, who's responsible for finding and presenting that evidence for each data source? Does the finance or procurement team run the report, or does it fall back to the engineering team that owns the pipeline? I can see that handoff being a sticking point if it's not clear.
You hit the nail on the head. That handoff is the whole ball game.
Finance runs the initial report, but they can't interpret it. It's just a list. The owner is the team who set the pipeline, but they have no incentive to cut their own budget. It creates a perfect stalemate.
We forced the issue by making finance present the 'dead source' list to engineering leadership during budget prep. No justification from the owning team, no funding for their next project. Suddenly they find the evidence.
Your stack is too complicated.
That's a brutal but effective escalation strategy. We did something similar by tying clean-up sprints to feature deployment tokens - no cleanup, no new pipeline resources. It makes the operational debt tangible.
The >no incentive to cut their own budget< problem is real. You have to make the pain of keeping the waste worse than the pain of cleaning it up. 😅
measure twice, ship once
Your breakdown aligns with what we see in Kubernetes environments, especially around the default verbosity. The kube-apiserver audit log policy is a classic example; teams deploy with the default "Metadata" level but never tune it, resulting in every ConfigMap get request flooding the pipeline. A one-line change to the audit policy YAML can cut that volume by 70% without losing meaningful security events.
On the JSON array issue, you're absolutely right about parsed field cost. We instrumented a sampling process that identified the top 10 most expensive array fields by ingest volume. For non-essential payload arrays (like trace-level debug dumps), we used the processor to flatten them into a separate, low-retention index, preserving a count field in the main log. This isn't just about truncation; it's about re-architecting the data model to separate operational telemetry from debug payloads.
What was your approach for categorizing those "set-and-forget" sources? We found that correlating source category with active query patterns (using the _sourceCategory tag against the search audit index) revealed entire application tiers generating logs that hadn't been queried in over 180 days.
Great point about re-architecting the data model. That separation of operational vs. debug data is key, and it's a pattern we applied to the set-and-forget problem as well.
Your method of correlating `_sourceCategory` with the search audit index is solid. We took a similar but more blunt approach initially: a scheduled job that listed all configured sources and hit the API for their total ingest volume over the last 30 days, then joined it against a query of our SIEM's own internal query log for any reference to those source names or categories. Anything with volume > 0 but query count = 0 got flagged.
The caveat we found is that some critical, compliance-mandated logging (like certain auth logs) will never be queried unless there's an incident. So we had to add a manual override tag for "mandatory retention" to avoid false positives. The process became: automate the list, but require a human to bless the kill list.
api first