What data pipeline actually works for a 200-user shop on Azure?

I’ve spent the last quarter evaluating data pipeline solutions for our Azure-hosted environment, which supports approximately 200 internal users. Our primary needs revolve around observability: collecting application logs from AKS, Azure SQL diagnostic settings, and custom application metrics, with the goal of routing this data to Datadog for APM and Grafana for internal dashboards. The volume is modest—under 50 GB/day—but the complexity lies in the need for reliable, cost-effective filtering and routing without introducing operational overhead.

The canonical suggestion for an Azure shop is, of course, Azure Monitor and Data Collection Rules paired with Event Hubs. However, in practice, I found this stack insufficient for nuanced data management. The lack of real-time processing flexibility before data egresses to a third-party SaaS like Datadog became a significant cost driver. We were essentially paying to ingest and export data we didn't need. This led me to evaluate dedicated pipeline tools, with Cribl Stream being a prominent candidate.

My testing criteria focused on:
- **Azure Native Integration:** Authentication via Managed Identity, ingestion from Event Hubs, and output to Azure Blob for archival, as well as to Datadog and Grafana Cloud Loki.
- **Processing Overhead:** The ability to drop verbose debug logs, redact PII from application logs, and condense multiple log lines into single events before they incur SaaS costs.
- **Operational Simplicity:** For a team of two SREs, we cannot manage a distributed cluster of workers. A single, robust node is preferable.
- **Cost Predictability:** The total cost of ownership, including the pipeline tool itself, must be lower than the savings it provides on the observability backend.

I deployed Cribl Stream on a single D4s v3 Azure VM (4 vCPUs, 16 GB memory). The setup was straightforward, and the Managed Identity configuration worked flawlessly for accessing Event Hubs and Blob Storage. I created a pipeline that demonstrated Cribl's strength: using Regex and JSON parsers to classify data, then applying conditional routes. For example, dropping `DEBUG`-level logs from non-production namespaces and sampling verbose health-check pings at 10% reduced our Datadog-bound volume by an estimated 35%.

However, the operational model gave me pause. While the single worker node handled our throughput, its high availability configuration would require additional nodes and a shared configuration persistence layer (like Azure Database for PostgreSQL), introducing more moving parts and doubling the cost. Furthermore, for our modest data volume, Cribl's licensing cost—even for the limited Free tier—felt disproportionate when considering the VM and managed database expenses on top. It is a powerful tool, but its architecture seems optimized for larger, multi-destination enterprises.

Consequently, I am now revisiting a simpler stack: Fluent Bit daemonsets within our AKS clusters, outputting directly to Datadog and Grafana, with its native filtering and rewriting capabilities. For Azure platform logs, Event Hubs with a small Azure Functions app to perform basic filtering and forwarding might suffice. This approach appears more aligned with a 200-user shop's scale and operational capacity.

I am keen to hear from others in similar-sized Azure environments. Has anyone successfully implemented a minimal, yet resilient, data pipeline that reduces observability costs without creating a management burden? Specific insights on Fluent Bit vs. Cribl for sub-100 GB/day, or alternative tools like Vector or OpenTelemetry Collector in this context, would be invaluable.

— Billy