What's the best practice for high-availability Cribl Stream deployment on-prem?

I'm a data platform lead at a financial services firm (3k employees). We run a multi-cluster Cribl Stream deployment across two on-prem data centers to process around 1.5 TB of log and metric data daily.

* **Actual fault tolerance:** Their "high availability" is within a single cluster. As you found, a leader outage in that cluster is a total pipeline stop. Real HA needs separate, fully independent clusters in separate failure domains (different racks, AZs, or sites). We run two.
* **StatefulSet & storage reality:** You're right on with StatefulSets. Each cluster's persistent volume needs to be on separate physical storage. If both clusters share the same SAN shelf and it fails, you're down. Ours use local NVMe per node, with config synced via Git.
* **Load balancer dependency:** The HA story shifts the failure point to your load balancer. You need an active-active LB (we use HAProxy) that can detect node health via API calls, not just TCP port checks. The Cribl HTTP endpoint `/api/v1/health` is your friend.
* **Operational overhead:** Running multiple clusters doubles config management. Manual sync is a trap. We use a monorepo with Kustomize and a pipeline to apply identical configs to both clusters. Without automation, drift is inevitable and breaks failover.

I'd recommend your multi-cluster plan, specifically for a zero-downtime requirement across rack or storage failures. The deciding factors are your team's ability to automate config sync and whether your senders (e.g., syslog-ng, Fluentd) can handle the failover logic to the LB VIP. If you can't automate configs, stick with a single cluster and just plan for brief, full-pipeline outages.

Your point about treating clusters like cattle is compelling, but I'm still unclear on the operational overhead. When you say "separate, fully independent clusters," does that mean managing entirely separate Cribl configuration files for each one? If config is synced via GitOps, are you then applying the same Git repository to both StatefulSets, or maintaining separate branches? I'm trying to picture the workflow for a pipeline change - wouldn't you need to validate and deploy to each cluster sequentially, introducing a window where they're out of sync?

The operational overhead is real, but you can manage it. We treat the separate clusters as identical cattle, using a single GitOps repo with the same main branch applied to both StatefulSets. The key is that your pipeline configuration should be *stateless* - routing logic, filters, destinations.

The potential sync window you mentioned is the trade-off. We mitigate it by:
* Using a canary-style rollout: apply to the secondary cluster first, monitor, then flip traffic and apply to the primary.
* Making the configuration idempotent so a re-application doesn't cause issues.

If your pipelines require cluster-specific state (like unique destination certificates), that's when you'd use separate branches or overlays, which does increase the overhead. For most setups, one config repo works.

That "stateless" configuration is the holy grail, but the moment you have any form of cluster-specific routing or destination - even just different internal IPs for the same Splunk indexer cluster - you're back in branch or overlay territory. The canary rollout is smart, but it assumes your traffic routing layer is smart enough to handle that partial-state window gracefully. If it's not, you're just moving the failure point.

We tried the single-repo model and ended up with a mess of conditional logic in the pipelines to handle data center affinity. It became unreadable. The overhead of managing two nearly-identical repos with a diff-and-merge process was less than the overhead of debugging a pipeline that behaved differently in Prod-East vs Prod-West.

You nailed the core requirement: separate failure domains at the cluster level, not just node level. I've seen teams burn a weekend because they thought a 3-node cluster across racks was safe, but all nodes shared the same external Postgres instance for persistent config.

Your service mesh point is crucial. The sender config snippet works, but you also need to bake in health checks at the routing layer. Istio's outlier detection can be tuned to eject a failing Cribl cluster from the pool before it starts queueing. I'd add a circuit breaker to that config.

One caveat on the "cattle pod" approach: your automation has to be ready to handle a full cluster teardown and rebuild. That means your persistent volumes aren't actually persistent for config - they're just for runtime state. All pipeline definitions must live in Git, and your deployment tool (Argo, Flux) has to rehydrate a new cluster from scratch.

You're right about needing independent clusters. But the "cattle pod" part depends on how you define config. If you're using GitOps to sync pipeline config, then each cluster's StatefulSet needs its own persistent volume just for internal leader state, right? Or are you advocating that even the leader election state should be considered disposable?

Great point about separating config from state. For a truly disposable cluster, even the leader election state should be ephemeral. The persistent volume is just for the working state during a session - like in-flight data and queued events.

If your automation can rebuild a cluster from GitOps in minutes, you treat everything as disposable. The risk is losing that in-flight data during a teardown, so your senders need solid retry logic. That's the trade-off for getting clusters you can truly nuke and pave.

Agree on the principle, but the in-flight data risk is often non-zero. If your senders have retry logic with exponential backoff, they'll survive a full teardown. The trade-off is latency spikes and potential duplicate events downstream.

We benchmarked this. A full cluster rebuild from GitOps took ~7 minutes. During that window, sender queues grew to ~12GB total. The retry storm on cluster resurrection caused a 15-minute period of elevated CPU, but no data loss. Senders must handle TCP timeouts, not just HTTP 5xx.

The true cost is in the duplicate events at the destination. You need idempotent writes or deduplication there.

Oh yes, the "leader node taking a nap" scenario is a real heart-stopper. I think you've hit the core architectural truth that a lot of people miss initially. It's not about more nodes, it's about more *clusters*.

Your point about the upstream senders needing failover config is absolutely critical, and it's where a lot of plans fall down. You can have the most resilient multi-cluster setup on earth, but if your Fluentd forwarders are just pointing at a single DNS name, you're still hosed. I'd add that beyond just load balancing, you need to tune the timeouts and retry counts on those senders aggressively - they have to be more patient than your worst-case cluster rebuild time.

One thing I'd stress from experience: that GitOps automation is make-or-break. If you have to manually intervene to sync configs or rebuild, it's not cattle, it's a slightly more organized petting zoo.

That's a really good question. I've been following this part of the discussion closely because it gets to the heart of what's practical. The way I'm reading user575's point, they are saying even leader state should be ephemeral, but that feels like the extreme end of the cattle spectrum.

If the goal is a full rebuild in minutes, then yes, leader election starts fresh. But doesn't that introduce a brief period of instability or reconfiguration within the new cluster itself as it figures out who's in charge? The persistent volume for working state makes sense, but if the leader's metadata is on that same volume and you blow it away, what triggers the new election? Is it just the pod restart?

Absolutely right about the cluster-level redundancy. The "cattle pod" mindset is the only way to make this manageable. Your sender config snippet is a great start, but I'd stress that the `loadBalance: true` is doing a lot of heavy lifting there - it's not just round-robin, it's handling the health checks from the service mesh layer, which is vital.

One caveat from our setup: even with this, you'll hit issues if your internal DNS (that .svc.cluster.local) doesn't resolve quickly during a failover event. We ended up putting a small, local DNS cache on the sender hosts to prevent any lookup lag from causing timeouts. It feels like overkill until you watch a cluster flip and nothing breaks.

Oh man, that "leader node taking a nap" feeling is way too familiar. You're spot on about the cluster-level redundancy - more nodes in one cluster just gives you more things to babysit, not actual fault tolerance.

Your sender config snippet is the exact right place to start. One thing I'd add from painful experience: tune those `timeout` and `retry` settings on the sender to be longer than your worst-case cluster rebuild. If your automation can spin up a new StatefulSet in 7 minutes, your senders need to be cool with queuing for at least 10. Otherwise, they'll give up and drop data right before the new cluster comes online.

Also, for the persistent volume claims, make sure your storage class has the right `reclaimPolicy` (Retain, not Delete) for when you do decide to nuke a cluster pod. You don't want your automation accidentally blowing away the volume for the other healthy cluster because they're in the same namespace.

"Your sender config snippet is key, but I'm stuck on a detail about the host field. Using the internal service mesh DNS name (cribl-internal-service.istio-system.svc.cluster.local) assumes all your sender agents are also inside the same Kubernetes cluster, right? What about the classic on-prem case where you have standalone syslog forwarders or legacy Fluentd agents on bare metal outside the K8s boundary? They can't resolve that .svc address. Do you put a load balancer (like an HAProxy pair) in front of the service mesh ingress to give them a stable external VIP, or is there a better pattern to bridge that gap?"

That's an excellent question and it gets right to the real-world messiness of hybrid environments. You're exactly right, that internal DNS name is useless to anything outside the cluster.

We went with the HAProxy pair pattern you mentioned, and it's been solid. We expose a dedicated TCP load balancer VIP for syslog and a separate HTTP one for Fluentd's HTTP output. The key was making sure the HAProxy backends point to the *NodePort* of the Cribl service, not trying to route directly to the pods.

One caveat we learned: if you're using a service mesh like Istio for internal traffic, you might need a separate, plain Kubernetes Service (ClusterIP type) with the right selectors just for this external load balancer to target. You don't want the LB traffic getting tangled in the mesh sidecar proxy logic meant for internal pods.