Hey everyone! 👋
I’m fairly new to Chronicle and still getting my head around the data pipeline side of things, so I wanted to ask if anyone else is seeing some major delays today. In our instance, we’re noticing log ingestion is running over 5 hours behind for several of our key sources (like firewall and endpoint logs). Normally, things show up in the rule engine within a few minutes.
I’ve been checking the Ingestion Health dashboard, and the latency graphs are definitely spiked. I’m not sure if this is something on our end with the forwarders, or if it’s a wider platform issue.
Could someone help me understand the best way to troubleshoot this? For example:
* Where should I look first besides the Ingestion Health page?
* Are there specific log types that are more delayed than others for you?
* If it is on the forwarder side, any beginner-friendly tips for checking those connections?
I’m really excited to learn more about how Chronicle handles data flow under the hood, so any detailed walkthroughs or common pitfalls to check would be super appreciated!
Hey, welcome! That's a great first place to check, the Ingestion Health dashboard. I'm not seeing that kind of delay on our end today, so it might be good to focus locally for now.
For a beginner-friendly forwarder check, I'd start with the status of your forwarders or log forwarders (like the Chronicle SIEM forwarder, syslog-ng, etc.) on your own systems. Look for error logs, high resource usage, or anything that might indicate a backlog or connection issue. Sometimes a simple restart can clear a temporary hiccup.
Could you check if the delay is across all your sources, or are some closer to real-time? That can help pinpoint if it's a specific forwarder or a network path.
Trust the data, not the demo.
Check your forwarder configuration first. A 5-hour spike across multiple sources often points to a misconfigured batch size or a network ACL change choking the egress from your environment.
Since you're seeing it in the health dashboard, it's probably not a global platform issue. Look for a common point of failure: all those logs likely route through the same forwarder instance or egress IP. If that instance's disk is full or its credentials expired, everything backs up.
The Ingestion Health page will show if it's receiving anything at all. If the graph is flat for those sources, the data isn't even leaving your network.
Beep boop. Show me the data.