Hey folks, been deep in the weeds on our Kubernetes log pipeline this quarter and hit a classic crossroads. We've been using Datadog for everything – APM, metrics, logs – but that ingest bill for container logs was getting *painful*. We started evaluating Cribl Stream to get control before data even leaves our VPC.
The core question for us: Do we stay all-in on Datadog's integrated experience, or introduce Cribl as a smart buffer to reduce cost and add flexibility?
Here's our current setup and the pain point. We run about 50 EKS clusters (mix of prod and dev). Datadog's agent is daemonset-based, forwarding all container stdout/stderr and pod/node logs. The volume, especially from debug logging in dev clusters, is huge. We were paying for a lot of noise.
With Cribl, we're testing a pipeline that sits between the FluentBit collection and Datadog. The idea is to filter and reduce aggressively before egress. Simple example of a Cribl drop rule we're testing for dev namespaces:
```yaml
# In Cribl Pipeline - Drop DEBUG logs from specific dev namespaces
filter: (kubernetes.namespace_name == "dev-api") && (level == "DEBUG")
# Then route the rest to Datadog
```
This alone looks promising for cost. But it adds another component to manage (Cribl workers in our cluster). The trade-off is operational complexity vs. potential 30-40% ingest savings we're projecting.
Has anyone else run this hybrid setup? I'm particularly curious about:
- Managing Cribl's routing rules at scale across many clusters.
- Whether you lose any valuable Datadog integrations (like log->trace correlation) when you pre-process logs with Cribl.
- If the savings were worth the extra hop, or if we'd be better off just tuning Datadog's agent configuration and sampling more aggressively within Datadog itself.
Love the Datadog UI and correlation features, but the FinOps part of my brain is screaming for control.
cost first, then scale
That makes sense, especially filtering DEBUG logs from dev. Did you run into any issues with the sampling itself, like accidentally dropping something you needed to keep? I'm always nervous about those filters being too aggressive.
Also, how's the added complexity with Cribl? Managing another component for 50 clusters sounds heavy.
That Cribl drop rule is a solid start. I'd push you to consider the retention angle, too - can you drop it entirely, or do you need a cheap, searchable copy for compliance? We keep a separate Loki instance for filtered dev logs, just in case.
Managing Cribl for 50 clusters is a real ops lift, though. Did you settle on a single distributed Cribl deployment or are you planning an instance per cluster? The single deployment becomes a new SPoF.
Sleep is for the weak