Having recently completed a multi-data-center Cribl Stream deployment for a financial services client with stringent 99.99% availability requirements, I've solidified a set of practices that move beyond the documentation's basics. The core challenge isn't just keeping the service running, but ensuring zero data loss, predictable failover, and manageable operational overhead in a self-managed environment.
The architecture hinges on a **active-active Worker Group** model across physically separate failure domains. Simply put, you run identical Worker Nodes in at least two locations (or availability zones), with a load balancer distributing events to the entire group. This is superior to active-passive for on-prem HA because it utilizes all infrastructure continuously and provides inherent load distribution.
### Critical Configuration Pillars
* **Persistent Queues (PQ) are Non-Negotiable:** This is your primary data durability mechanism. PQ must be configured on a highly available, low-latency shared storage layer (e.g., a SAN, or a replicated NFS/CIFS cluster accessible from all nodes). The queue path must be identical on all workers.
```yaml
# Example from worker's cribl/state/cribl.yml
persistentQueue:
enabled: true
destType: disk
dataPath: /mnt/cribl_pq/data
commitPath: /mnt/cribl_pq/commit
pqSizeMB: 1000000
```
* **Load Balancer Intelligence:** Your load balancer (F5, HAProxy, nginx) must perform **TCP-level health checks** on the Worker Node's configured data port (e.g., 9200). It must immediately stop routing to a failed node. Session persistence (sticky sessions) is **not** required and can be detrimental to load distribution.
* **Configuration Management:** All Worker Nodes must share identical configuration. This is achieved by pointing all nodes to the same **Leader Node**. The Leader itself should be made HA using Cribl's embedded Leader HA or, for maximum control, an external distributed configuration store (like etcd) with the `CRIBL_DIST_CONFIG` environment variable.
* **Observability of the Deployment Itself:** You cannot manage what you cannot measure. Deploy Cribl's own metrics to a dedicated monitoring stack (Prometheus/Grafana recommended). Key alerts should be on:
* Worker Node health status (from the Leader's perspective)
* Persistent Queue depth and disk usage
* Output destination health and connection latency
### Common Pitfalls & Mitigations
* **Storage as a Single Point of Failure:** The shared storage for PQ is critical. If it goes down, all nodes fail. Invest in a fault-tolerant storage solution with synchronous replication and monitor it aggressively.
* **Network Partition (Split-Brain):** In an active-active setup, a network partition can cause nodes to lose contact with the Leader. Ensure your network infrastructure provides redundant paths and consider tunable timeouts for configuration synchronization.
* **Cost of Over-Provisioning:** HA inherently requires more resources. Use capacity planning based on your peak throughput with headroom for a single-node failure. Monitor license usage across the expanded node count to avoid surprises.
The ultimate test is a **controlled failure simulation**. Schedule a drill where you forcibly terminate a Worker Node and observe: does the LB detect it within seconds? Does PQ on the surviving nodes absorb the load? Do your outputs handle the slight surge? This validation is more valuable than any diagram.
I'm curious—for those running similar setups, how have you approached the shared storage challenge, and what failure scenarios have you encountered that your design successfully absorbed (or didn't)?
—N
Latency is the enemy.
I'm a senior architect at a large healthcare network, managing log ingestion from about 50k sources. We've been running a self-managed Cribl Stream deployment across two data centers for over two years, handling around 2.5 TB/day, and our practices align closely with what you've described. Our setup is also active-active with a focus on zero data loss.
Here's a breakdown of what's critical from an operational perspective:
**Failover Validation Frequency:** The most common gap I see is testing. We schedule a full failover test quarterly, pulling one entire site offline during a maintenance window. We found and fixed a DNS caching issue that would have caused a 90-second data brownout.
**Shared Storage Latency Impact:** You called out low-latency storage for Persistent Queues. This is vital. In our initial proof of concept, placing the PQ on a high-availability NFS share with ~15ms latency caused a 30% throughput drop compared to local SSDs. We had to work with storage teams to get that under 5ms.
**Load Balancer Session Persistence:** For TCP-based sources (like Syslog), you must enable session persistence (sticky sessions) on your LB so a data stream hits the same worker for its duration. We lost about 0.1% of events from flapping before we turned this on.
**Resource Overprovisioning:** Our production nodes are sized at 50% over the documented requirements for CPU and memory. This headroom absorbs traffic spikes during a failover event when one site takes the full load, and prevented performance degradation the two times we've had to fail over for real.
Your approach is solid. Given your 99.99% requirement, I'd recommend your architecture but would ask about your monitoring for queue depth and node health, and how you're handling configuration drift between the two sites. That's where we've spent the most operational time.
Keep it real.
That's a fantastic real-world point about load balancer session persistence for TCP streams. It's a subtle detail that can quietly break event ordering or cause session state issues if you're using a stateful processing function.
Building on your operational focus, we added automated health checks to our load balancer that go beyond simple TCP connectivity. They run a small diagnostic API call to each worker node and verify it can actually process a test event through our critical pipelines. This caught a few edge cases where a node was "up" but silently failing on specific routes due to a config sync hiccup.
Your quarterly full-site failover test is so important. We do bi-annual tests but run a "chaos" exercise monthly where we randomly drain one worker node at a time during peak hours. It's stressful but really validates the resilience of the remaining nodes and the load balancer's reaction time.
migrate with care
Active-active across data centers makes sense for full infrastructure use. But what about the cost of that shared storage layer for the queues? Is the SAN or replicated setup you mentioned a significant budget factor compared to the rest of the deployment? I'm trying to understand the total overhead.
Active-active over multiple DCs is clear. But what's the minimum realistic latency for that shared PQ storage? At what point does the lag cause the queue to back up for high-volume sources?
You mention SAN or replicated NFS. Is there a threshold, like 5ms vs 20ms, where you'd have to adjust the PQ buffer sizes or reconsider the architecture?
null
That's a super practical question, and we ran into exactly that during our performance testing. We aimed for sub-5ms latency for our PQs to keep up with our main event firehose. In our real-world setup, we started seeing noticeable backpressure on the busiest pipelines once latency crept above about 8ms. It wasn't a brick wall, but you could see the queues growing faster than they were draining during bursts.
So yes, you absolutely have to adjust buffer sizes and maybe even pipeline design with higher latency. We had to increase the memory buffer for a few high-volume sources and tweak our worker group's `pqMaxSize` when we knew we were crossing the 10ms threshold to a DR site. If you're consistently above 15-20ms, I'd really question if active-active with shared storage is the right fit, or if you'd be better with an active-passive model and a different replication strategy. It becomes a tuning game between buffer memory and your risk tolerance for data lag.
What's your typical volume per source? That's the other big factor in figuring out your latency threshold.
Data is the new oil
The latency/volume tradeoff is real, but everyone ignores the real cost driver: that shared storage. Aiming for sub-5ms usually means a high-end SAN or all-flash array. That's not just a capital expense, it's a permanent vendor anchor around your neck.
You can tweak `pqMaxSize` all day, but that just shifts the cost from storage performance to memory overprovisioning on the worker nodes. The budget discussion always gets buried after the architecture slide.
What's the actual price tag on that 8ms versus 15ms storage tier in your environment? That's the threshold that matters more than the technical one.
-- cost first
Your point about active-active utilizing all infrastructure is architecturally sound, but I'd stress the need for rigorous capacity planning to realize that benefit. If your combined worker nodes across DCs can't handle the full load with at least one site down, you haven't actually achieved the inherent load distribution you mentioned; you've just built a more expensive active-passive system. We model for N+2 capacity, ensuring the surviving site can absorb the traffic from a complete DC failure without hitting memory or CPU saturation, which is a different and more expensive calculation than simple N+1.
The shared storage requirement for PQs is the linchpin, and your low-latency callout is correct. However, I've seen teams underestimate the I/O pattern. It's not just latency; it's sustained write throughput and IOPS during a surge. A SAN might promise 5ms latency under test but fall to 50ms during a concurrent backup or if the replication link gets saturated. You need to provision and monitor for the worst-case ingest spike, not the average.
--perf