I have been operating a Cribl Stream deployment for several months now, primarily to unify log collection from various Kubernetes clusters and forward processed data to our analytics pipeline in BigQuery. The setup has been largely stable, but I am now encountering a perplexing issue: one of my worker groups is restarting at seemingly random intervals, approximately every 4 to 12 hours. The restarts are not correlated with any observable load spike or configuration change on our part.
Upon investigating, I find the Cribl logs themselves to be conspicuously unhelpful. The `cribl.log` shows only normal operational messages prior to the restart, followed by a standard startup sequence. There are no error traces, no out-of-memory warnings, and no evident process crashes. The system metrics available within the Stream UI do not show any resource exhaustion (CPU, memory, or file descriptors) leading up to the event. This complete lack of signal in the primary log stream is the core of my frustration.
My current environment and configuration details are as follows:
* **Cribl Stream Version:** 4.1.3
* **Deployment Model:** Distributed, running on a dedicated AWS EC2 instance (c5.2xlarge).
* **Worker Group Size:** 3 worker nodes.
* **Primary Sources:** HTTP/S, Splunk HEC, and a File Monitor for tailing local logs.
* **Destinations:** Amazon S3 (for raw archives), and a custom HTTP sender pushing to a Dataflow job that loads into BigQuery.
* **Key Pipeline Complexity:** We are using several ETL-like functions, including:
* Parsing nested JSON structures.
* Masking specific PII fields (using the Mask function).
* Conditional routing based on log source.
* A significant number of `eval` expressions for field derivation.
I have already performed standard diagnostics:
* Verified disk space and inodes on the host.
* Confirmed no system-level OOM killer events via `dmesg`.
* Checked for any competing processes or cron jobs that might interfere.
* Reviewed the Linux system logs (`/var/log/messages`, `journalctl`) around the restart times, which also show nothing.
My hypothesis is that the issue may be tied to a subtle resource leak or a specific, edge-case event within a pipeline that causes a worker process to become unresponsive, triggering a restart by the master. However, without any definitive logging, I am left to speculate.
I am seeking guidance from the community on deeper diagnostic avenues. Specifically:
* Are there internal, more verbose Cribl logs or metrics that are not exposed by default in the UI?
* Could certain pipeline configurations (e.g., recursive evals, overly large lookup tables) lead to silent failures?
* What are the best practices for monitoring the health of worker processes at the OS level, beyond standard `top` or `htop`?
* Has anyone encountered similar "silent restarts" and traced them to a particular function or source type?
I am prepared to instrument the host more thoroughly but need direction on what to look for. The randomness of the event makes it difficult to catch in the act, but the operational instability is becoming a concern for our data pipeline's reliability.
Extract, transform, trust