Hey everyone! I've been wrestling with a cost control problem in my observability setup, and I'm hoping some of you have navigated this before. We're using a major platform (you know the one, with the three letters) and our bill is primarily driven by log and trace ingest. The pricing model is pay-as-you-go, which is fantastic for flexibility but terrifying during unexpected traffic spikes.
We had an incident last month where a third-party service went haywire and started logging its entire state every few milliseconds. It wasn't a critical service for us, but the log volume exploded and we got a bill that made our finance team do a double-take. We fixed the root cause, but it got me thinking: I need a safety net.
Here's my dilemma and what I've explored so far:
* I want to set a **hard daily cap** on ingest volume to prevent bill shock.
* However, I **don't** want to just shut off all data collection once we hit that cap. That feels like throwing the baby out with the bathwater—critical errors from our core application could be lost.
* I'm looking for a smarter way to "throttle" or "sample down" once we approach our daily budget, prioritizing what matters most.
My current stack handles routing and sampling logic at the collector level (using OpenTelemetry collectors). I've been digging into processors like `probabilistic_sampler` and `filter`, but they seem to operate on a per-event basis, not against a rolling daily volume total.
So my question for the community is twofold:
1. **Is there a native feature within any of the major observability platforms (Datadog, New Relic, Splunk, etc.) that allows you to set a daily ingest budget and then automatically apply sampling or filtering rules once you're, say, at 90% of that budget?** I haven't found one yet, but I might be searching for the wrong terms.
2. **If the platform doesn't offer it, has anyone built a custom control loop?** I'm imagining something that:
* Monitors the platform's metrics API for your daily ingest volume.
* At certain thresholds, triggers an automation (via webhook) to reconfigure your forwarder (e.g., Otel collector) to apply more aggressive sampling or to drop logs from certain low-priority services.
* Resets the configuration at the start of the next day.
I'm very comfortable with the automation piece—it's similar to lead routing logic in my marketing stack—but I'm unsure if this is a solved problem or if I'm over-engineering it. Maybe there's a simpler agent-level configuration I'm missing?
I'd love to hear what strategies you all are using, especially if you've managed to balance cost predictability with data fidelity. Detailed war stories and even snippets of your collector configuration logic would be incredibly helpful!
Test everything, trust nothing