Hey everyone! 👋 I've been wrestling with a timing issue in our Graylog setup for the past week and could use some collective wisdom.
We're using Graylog v5.1 as our SIEM for some custom application monitoring. The raw logs (JSON over GELF TCP) have precise nanosecond timestamps from the application itself. However, when an alert rule triggersβsay, for a specific error patternβthe alert timestamp in Graylog is sometimes **5 to 15 minutes later** than the actual log event timestamp. This delay is making our incident response feel sluggish, even though the logs are ingested almost instantly.
Here's a simplified version of our alert condition:
```json
{
"expression": {
"expr": "some_field == "ERROR"",
"right": null,
"type": "expr"
},
"repeat_notifications": false,
"backlog_size": 50
}
```
What I've checked so far:
* The source logs have a valid `timestamp` field.
* The Graylog pipeline for this stream doesn't modify the timestamp.
* The system clocks between the log source, Graylog server, and my browser are all in sync (NTP is working).
* The alert is set to check every minute (`check every 1 minute`).
My hunch is it might be related to message processing buffering or perhaps the alert scheduler? Has anyone else run into this and found a reliable fix? I'm especially curious if there are any internal queue settings or if the alert condition evaluation respects the event timestamp or purely the system time when it scans.
Any pointers on where to look next would be amazing. I can share more config snippets if needed!
Clean code is not an option, it's a sanity measure.