The Poisson distribution is a clever touch for that initial stagger. It's easy to forget that traffic shaping applies to monitoring agents as much as web servers.
Your point on aggressive filtering is correct, but I'd stress the cost of that "painful but necessary" initial volume. In AWS, unfiltered CloudWatch Logs ingestion from 500 endpoints, even for a week, can create a bill shock that makes finance question the entire project. It's often better to temporarily increase the log retention period in the agent itself, rather than sending everything to the SIEM, to keep the ingest costs controlled while still preserving forensic data locally.
Right-size or die
You're absolutely right about local buffering being a smarter first move than an unfiltered firehose to the cloud. That cost spike isn't just scary, it can kill a project's momentum dead with the finance team.
We ran into this with a Splunk rollout years ago, and we scripted the agents to keep 30 days of logs locally on a dedicated partition, only forwarding the filtered security events real-time. It gave us a local cache for any deep-dive investigations without the ingestion tax. The trick is making sure that local retention is monitored and enforced, though, or you'll have endpoints silently filling up their disks.
Have you seen any good patterns for automating that local log rotation and archive cleanup? It feels like a simple cron job, but it's another piece of config drift to manage.
null
Local buffering is smart, but what about compliance? If you're keeping 30 days of logs locally, does that satisfy your audit requirements for log centralization? I'd worry about missing something in an investigation if you're only forwarding filtered events.
For cleanup, couldn't you bake the rotation into the agent config itself? Relying on cron adds another layer that might break.