We were paying $80k/month for Splunk ingest. Now paying $68k. The difference is a Cribl pipeline that drops low-value logs before they hit the license meter.
Goal: Keep security and app error events, discard generic noise. Our filter logic uses a combination of regex and JSON pathing.
Key filters applied in order:
* Drop all `DEBUG` and `INFO` level logs from our app containers, except those tagged with `security_audit`.
* Drop all Kubernetes node heartbeat logs (`node-monitor`).
* Drop specific chatty middleware health checks (`/health` endpoints returning 200).
* Sample verbose network proxy logs at 10% (reducing volume, keeping a sample for traceability).
The core function for our app logs:
```
// Filter for container logs, prior to Splunk HEC
if (this.source == "k8s_container" && this.log.includes("DEBUG") && !this.tags.includes("security_audit")) {
drop();
}
```
Result: 15% reduction in ingest with no impact on critical alerts or dashboards. ROI on the Cribl instance was under 30 days.
ea
Prove it with a benchmark.
Nice breakdown! That ROI is insane, congrats.
Your point about sampling network logs at 10% is smart - I think a lot of teams are afraid to sample anything, but for traceability you really just need a representative slice.
One thing we added later was a rule to keep all logs from specific high-risk users or beta testers, even if they're INFO level. It adds a tiny bit back, but gives product a clean funnel for troubleshooting those key segments. Might be useful if you do any targeted releases.
ABT – always be testing
Agreeing to sample logs is the easy part. The hard part is getting product teams to actually define what a 'high-risk user' is. That list becomes political instantly, and then you're back to ingesting everything because no one wants their pet feature excluded. Good luck keeping that 'tiny bit' tiny.
If it sounds too good, read the release notes
That's a really good point about the politics. I hadn't considered that part yet. We're small enough that I just asked our product lead for a list of high-risk flags and he gave me three user properties to key off of. It's all technical right now, like `is_internal_user` or `has_beta_flag`.
I can see how that gets messy fast in a bigger org. Do you have any suggestions for keeping that definition objective? Maybe tying it to a specific permission level in the app, instead of a named list of people?