I've been running Panther in a multi-account AWS environment for about six months, primarily for real-time security alerting. While the performance is generally good, I've observed a consistent pattern where certain alerts—particularly those from CloudTrail rules—arrive in our Slack channel with a latency of 5 to 8 minutes from the timestamp of the actual event.
This is problematic for our FinOps and security response workflows, where a delay of even a few minutes can be significant. My initial assumption was that the delay was introduced by our log pipeline, but after checking, the CloudTrail events are being delivered to our S3 bucket well within 60 seconds.
I've reviewed the Panther documentation on alert delivery and haven't found a clear explanation for this buffer. My hypothesis centers on how Panther processes and batches events before evaluation. Could the delay be due to:
* The polling interval of the S3 log source?
* Internal queuing or batching within the Panther engine for cost-efficiency?
* A configuration artifact related to how we've defined our real-time alert rules?
For context, here’s a simplified version of the rule logic we're using, which triggers on specific IAM actions:
```
def rule(event):
return (event.get('eventSource') == 'iam.amazonaws.com' and
event.get('eventName') in ['CreateUser', 'CreateAccessKey'])
```
Has anyone else conducted latency analysis on their Panther alerting? I'm particularly interested in whether this is a known trade-off with the platform's architecture, or if there are tuning parameters I've missed. A 5+ minute delay seems to contradict the "real-time" classification.
—A
Every dollar counts.