Alright, has anyone else been drowning in Azure Diagnostic logs that Datadog just slurps up, costing a fortune and making it hard to see the actual signal? I finally got a Cribl Stream instance set up as a proxy to clean all that up before it hits Datadog, and the results have been game-changing for our bill and our dashboards.
The core idea is simple: route your Azure Diagnostic Event Hub through Cribl, filter/reshape, then send only what you need to Datadog. Here's the high-level flow I set up:
* **Source:** Azure Event Hub (using Cribl's built-in source). You'll need the connection string and hub name. It handles the Azure platform logs (Activity, Resource, etc.) seamlessly.
* **Key Processing Steps in the Pipeline:**
* Drop entire log categories you don't need (looking at you, `Administrative` logs for some services).
* Use `Eval` functions to filter out high-volume, low-value noise *within* log categories (e.g., specific routine health pings).
* Prune unnecessary fields from the bloated JSON payloads before forwarding. Azure logs come with a ton of metadata you often don't need in Datadog.
* Optionally, use Cribl to parse and extract custom metrics from logs if that's useful, sending them to Datadog metrics via the StatsD output.
* **Destination:** Datadog HTTP Logs intake API. Cribl's Datadog output is pre-configured, so you just need your API key.
The biggest win was using a simple `Eval` filter like `__eventHub.cg!="noisy_resource_group"` to drop entire resource streams at the source. Also, setting up a sample rate for extra verbose but occasionally useful logs saved a ton.
My main pitfall was initially being too aggressive and cutting logs we later needed for an audit trail. The solution was to use Cribl's built-in `Passthru` output to send a raw copy of *everything* to cheap storage (like an S3 bucket) as a backup, while only the filtered stream goes to Datadog. That way, you have the raw data if you need to re-index or investigate, without the premium Datadog storage cost.
If you're doing this, start by analyzing your current Datadog log volume by Azure resource type and category – it'll be obvious where to focus your filtering rules. It took about half a day to get the basic pipeline running and another day to fine-tune the filters based on what we actually needed for alerts and dashboards. The cost savings literally paid for the Cribl setup in the first month.
Ship fast, measure faster.