Everyone's dumping logs into Datadog or Splunk and paying for 30-day retention by default. That's insane. Most logs are useless after 48 hours.
Built a simple sidecar filter that runs alongside my logging agent. It identifies and exports low-value logs to S3/Glacier after 2 days, cutting my high-res log volume by ~70%. The expensive vendor only sees what's actually needed for active debugging.
Key filters applied:
* Health checks and load balancer pings
* High-volume, low-info debug logs from specific libraries
* Repeated connection spam from known internal sources
* Any log line matching a "noise" regex pattern we defined
Cost impact:
* Vendor observability bill down 65%
* S3/Glacier storage cost is negligible
* Still have full fidelity for critical errors and recent requests
If your team isn't doing this, you're burning money on storage you'll never query.
show me the bill
Finally, someone gets it. I've seen teams spend six figures on log platforms while their actual debugging needs could fit in a cheap S3 bucket with some grep scripts.
One caveat though: that "noise" regex pattern can bite you later. I once saw a team filter out "connection reset" logs, which later masked a slow-drip network partition because the errors weren't loud enough to be "critical." Make sure your filter logic is documented and reviewed when you add new services.
The real next step is pushing this upstream. Why are libraries emitting those high-volume, low-info debug logs in production by default? Turning that stuff off at the source saves you the filtering compute cost too.
keep it simple