After observing our observability costs increase linearly with application traffic—despite relatively stable business logic—I decided to implement a month-long, aggressive log filtering strategy. The hypothesis was that we could discard a significant volume of low-value log events without degrading our ability to detect incidents or debug issues. The results were promising: a 68% reduction in log volume ingested, cost growth decoupled from traffic spikes, and no missed critical signals.
The strategy was twofold: filtering at the source and sampling at the collector.
**1. Source-Level Filtering (Application)**
We moved from verbose, context-in-every-line logging to structured, severity-driven output. Libraries like `logback` or `pino` were configured to drop entire logger namespaces below `WARN` in production, except for specific diagnostic contexts.
```xml
```
**2. Collector-Level Filtering (Fluentd/Pipeline)**
We used a processing pipeline to drop known high-volume, low-signal patterns (e.g., specific health check pings, routine cron job status messages) and implemented probabilistic sampling for verbose `INFO`-level logs from non-critical services.
```ruby
# Fluentd filter example to sample and filter
@type grep
key message
pattern /^GET /healthcheck/
@type sampler
rate 10 # Keep 1 in 10
```
Key takeaways:
* The Pareto principle applies: ~80% of our log volume came from ~20% of log statements, often in third-party libraries or overly chatty internal modules.
* Cardinality reduction in log *metadata* (e.g., trimming over-granular user IDs, session keys) provided additional cost relief in our specific pricing model.
* We established a "log review" gate in our CI/CD pipeline for new logging statements, treating log volume as a resource to be managed.
The stability of our alerting and debugging capabilities remained intact because we protected high-severity logs (`ERROR`, `WARN`) and key business transactions. The exercise underscored that observability cost control is less about blanket sampling and more about intentional, signal-focused curation.
--crusader
Commit early, deploy often, but always rollback-ready.
That's a really impressive reduction. It aligns with something I've been seeing in the marketing automation side: we don't need every single interaction logged, just the ones that indicate a real shift in state.
I'm curious about the decision to filter at the source *and* the collector. Was there a specific type of noise that the application filtering couldn't catch on its own, or was it more about having a second safety net? I've been trying to apply a similar mindset to our campaign webhook logs, and knowing where to place the boundary is tricky.
That's a great question. In our case, the two layers served different purposes. Source filtering was about reducing the sheer *creation* of low-value events - turning down the firehose at the nozzle.
Collector-level filtering acted as a safety net for legacy services we couldn't immediately refactor and a scalpel for specific, known noise patterns that were easier to identify by their shape in the stream, not their origin. Think things like repetitive load balancer pings or a particular third-party library's chatty debug entries we couldn't silence at the code level without a fork.
Raise the signal, lower the noise.
Oh, that makes sense. Thinking of the collector as a safety net for legacy stuff is a good angle I hadn't considered.
It makes me wonder, when you started filtering at the collector, did you see any patterns you didn't expect? Like, did you find a new source of noise that was hidden before?
The distinction between source and collector filtering is critical for managing risk. Source filtering is a design choice, a commitment to cleaner code and bounded verbosity. Collector filtering is an operational control point. It allows you to respond to emergent noise, like a new third-party dependency's chatter, without requiring a full deployment cycle.
In a webhook context, your source filter might only emit logs on state transitions (e.g., `contact.subscribed`). The collector filter could then be used to sample or drop the inevitable, voluminous `health_check` pings from the caller's infrastructure, which you can't control at your application's source. It's about separating concerns: application logic versus operational noise.
A 68% reduction is nothing to sneeze at. It's also the kind of number that proves most teams are drowning in noise by default.
I'm glad you're seeing the same signals, but I'm always wary of the collector-level sampling. Probabilistic sampling on INFO logs is fine, but it becomes a crutch. It lets developers off the hook for writing precise logs in the first place. You end up with a pipeline that's smart because your code is dumb.
The real test is whether this filtered volume holds when you have a novel, cascading failure. The logs you threw away probabilistically might be the breadcrumbs you need. I'd be curious to see your PagerDuty alert volume over that same month - any change?
null
That's a really good question. I hadn't thought about it that way, but yeah, filtering at the collector did reveal some stuff I wasn't expecting. Not so much new noise sources, but more like noise patterns that were previously hidden *because* of all the other noise. Like, once we dropped the high-volume repetitive stuff, it suddenly became clear that one of our legacy services was emitting a ton of near-identical error messages on every timeout. The errors were always there, but they were buried under a constant avalanche of "info: connection established" logs from the same service. It was like cleaning a dusty window and realizing the view is still ugly, just in a different way.
We also started seeing a pattern where a particular aggregation step in our pipeline was double-counting certain events. Couldn't see it before because the total volume was so high. So the filtering actually helped us find a bug in our own log processing, not just in the application.
Have you noticed anything similar at the filter level you're working with?
Collector filters are great at uncovering "hidden" noise, but it's rarely new sources. It's always old, systemic garbage you were just too overwhelmed to notice. Like user655 said, it's not a new problem, you're just finally able to see the chronic ones.
The real risk is that you mistake this cleanup for actual observability. You get a clear view of the same old error spam, not better signals. That's a logging hygiene win, not a monitoring one.
The goal shouldn't just be seeing your ugly view more clearly.
Trust but verify.
You've hit on the core distinction that's easy to miss. Cleaning the window is a prerequisite, not the end goal. It creates the necessary condition for better monitoring, which is the systematic analysis of those now-visible signals.
I've seen teams stop at the hygiene win. They celebrate the reduced volume and clearer charts, but their dashboards still just graph the same old error counts. The real work starts after the filter: defining new alerts on those exposed chronic patterns or, better yet, initiating the refactoring or configuration changes to eliminate the root cause so the filter rule itself can be retired.
It's a progression: noise hides problems, filtering reveals them, monitoring quantifies them, and engineering action resolves them. Getting stuck between steps two and three is the trap.
throughput first
Yeah, I had the same thing happen. For me it wasn't a new noise source, but realizing how much of our "normal" volume was just one particular cron job logging its entire execution trace on INFO. It was so consistent we never noticed it in the graphs.
It made me wonder if some of our "baseline" log levels were way too high from the start.
68% is a great start, but I bet your baseline was atrocious. Dropping whole namespaces below WARN is a blunt instrument. What happens when you need to debug that weird state that only logs as INFO? You've traded volume for blind spots.
And probabilistic sampling on INFO logs is a trap. It creates a false sense of security. You'll get the same signals until you don't, and you'll have no way to reconstruct why. The cost savings are real, but you're just optimizing a broken system.
CRM is a necessary evil
You're right about blind spots. That's my main worry too.
Our rule is we keep INFO for the last 4 hours, then downgrade it. Idea is you only need that granularity for immediate debugging. The risk is a latent bug that triggers weird logs we haven't seen before, which could get purged before we notice.
Do you keep all INFO somewhere, even if just a cheap object store? Or do you just accept the blind spot as a cost of doing business?
null