Everyone's running around trying to fix the symptom instead of the disease. We've got entire subforums dedicated to cardinality reduction, log sampling, and trying to outsmart vendor pricing models. It's an entire cottage industry of coping with our own data hoarding.
The disease is simple: we log everything because we can, not because we should. Debug-level logs from production? Full HTTP request/response bodies for every API call? Three different telemetry systems emitting the same "user logged in" event? It's a reflex, not a design.
We're paying for this on multiple fronts:
* The direct vendor bill, which scales with volume and cardinality.
* The engineering hours spent building and maintaining these Rube Goldbergian filtering pipelines.
* The performance hit on our own applications generating and shipping all this noise.
* The cognitive load on engineers who now have to sift through a terabyte of logs to find the one useful line.
Instead of asking "how do we ship and store all this cheaper?" we should be asking:
* What is the *minimum* data required to confirm this service is healthy?
* What specific failure modes require forensic data, and can we trigger verbose logging only on those conditions?
* Do we really need to keep debug logs for 30 days, or is 48 hours sufficient?
* Have we actually *defined* what we're looking for in these logs, or are we just collecting "in case"?
This isn't about turning off observability. It's about intentional, surgical instrumentation. A single, well-structured, high-quality log line is worth a thousand noisy ones. The goal isn't to fill a data lake; it's to answer a question. Start by deleting log statements, not adding them.
—JP
If it's free, you're the product. If it's expensive, you're still the product.
You're right about the reflex logging. But the problem starts one level up, with the people asking for the dashboards.
Engineers log everything because product or sales asks, "Can we see how many users click this obscure button?" or "We need to track every step of this funnel for the board deck." Saying no means being the bottleneck, so we default to logging it all. The real question isn't just what we need for health, but who gets to decide what's necessary. Until that's fixed, the pipelines just get more creative.
Your CRM is lying to you.
Totally feel that. I saw this happen with a contract last year. Sales needed "granular usage data" for a potential upsell, but the spec was basically everything. It turned into a massive line item for log ingestion.
How do you push back without being the bottleneck? I always worry about the "just in case we need it later" argument.
Still learning.
The "just in case" argument is pure fantasy. It's an insurance policy paid for with immediate, real cash and complexity. That "granular usage data" sales wants? It's a bill, not a product.
Tell them you'll build it on demand when the contract is signed. Otherwise you're funding their fishing expedition with your infrastructure budget.
If you're the bottleneck for bad ideas, good. That's your job.
Keep it simple