I have been conducting a detailed performance and reliability analysis of Sumo Logic's collector infrastructure as part of a larger study on observability pipeline overhead. During these tests, I have encountered a particularly insidious failure mode that I believe warrants a thorough community discussion: the scenario where a Sumo Logic Collector (or Installed Collector/OpenTelemetry Collector) reports a **"Healthy" status in the Sumo Logic web interface, yet is demonstrably not ingesting or forwarding any log or metric data to the assigned source categories.**
This state creates a false sense of security and can lead to significant gaps in monitoring coverage. My initial hypothesis was a network egress issue, but subsequent controlled benchmarks revealed the problem can persist even with verified network connectivity. The root causes appear to be multifaceted and often interlinked.
Based on my systematic troubleshooting, I have isolated several potential failure points that can manifest this specific symptom. I will outline them in order of diagnostic priority, including the verification commands and configuration snippets used to identify each.
**Primary Investigation Areas:**
* **Collector Cache Backpressure:** The local collector cache (`$SUMOLOGIC/agentdata/`) can enter a state where it is not being drained due to transient upstream API errors or minor payload formatting issues. A "Healthy" heartbeat may continue while the actual data queue is stuck.
* **Diagnostic Command:** Monitor cache directory growth.
```bash
du -sh /usr/local/sumocollector/agentdata/
ls -la /usr/local/sumocollector/agentdata/ | wc -l
```
* **Check Collector logs for silent retries:**
```bash
grep -i "retrying|dropping|failed to send" /var/log/sumocollector.log
```
* **Source Configuration Mismatch:** The defined source `category` (e.g., `app/prod/nginx`) on the collector must exactly match the `_sourceCategory` query used in the Sumo Logic platform. A trailing space, case sensitivity for custom metadata, or a wildcard mismatch can result in data being sent but not appearing in expected views.
* **Verification Step:** Cross-reference the `json` configuration file on the collector host with the UI.
```json
// Example: /etc/sumologic/sources.json
{
"api.version": "v1",
"sources": [
{
"name": "My App Logs",
"category": "myapp/production",
"pathExpression": "/var/log/myapp/*.log",
"sourceType": "LocalFile"
}
]
}
```
* **File Permission or Rotation Issues (for File Sources):** The collector process user must have read permissions on the target log files. More subtly, if log rotation occurs with a move/rename pattern (rather than copytruncate), the collector can lose its file handle and fail to pick up the newly created file, even while the process health check passes.
* **Diagnostic Commands:**
```bash
sudo -u sumologic cat /var/log/myapp.log # Test permissions
lsof -p $(pgrep -f sumologic) | grep log # Check held file descriptors
```
* **HTTP Source or OTel Collector Buffer/Queue Saturation:** When using an HTTP Source or OpenTelemetry Collector configuration, in-memory queues (`sending_queue`) or batch processors may be silently discarding data due to misconfigured size limits or export timeouts that are not reflected in the basic health probe.
* **Relevant OTel Config Check:**
```yaml
exporters:
sumologic:
endpoint: "..."
sending_queue:
queue_size: 5000
enabled: true
service:
telemetry:
logs:
level: "debug" # Temporarily enable for investigation
```
I am currently designing a reproducible test bench to quantify the conditions under which each of these failures occurs. In the meantime, I would appreciate community input on the following:
* Has anyone else observed this "healthy but not sending" state, and if so, what was the ultimate root cause in your environment?
* Are there any known collector version-specific bugs (particularly with versions 19.245-22 and above) related to cache management or silent source disabling?
* What are the most effective logging configurations (`collector.log` verbosity, OTel `debug` settings) to surface these issues before they become critical?
* Beyond the points listed, are there any other systemic issues in the data pipeline—particularly related to collector registration token renewal or intermediate load balancer/proxy configurations—that could manifest these symptoms?
My goal is to synthesize findings into a deterministic troubleshooting flowchart and share benchmark results on collector behavior under various failure injections (disk I/O pressure, network latency spikes, upstream HTTP 429/503 responses).