Just finished a phased rollout of Cribl Stream to our retail store agents. About 100 endpoints, sending Windows event logs and POS app data to our SIEM. Goal was to filter noise and cut costs.
We hit two big snags. First, the Windows agent install via our RMM tool was smooth, but a few stores on flaky satellite links had persistent connection drops. The agent buffer filled up and we lost data until the next sync. Had to tweak the buffer settings per-site, which was manual.
Second, our pipeline to drop noisy local Windows events before the SIEM worked great, but it accidentally filtered out a critical security event ID that SecOps actually needed. Took a day to trace and fix the rule logic.
Anyone else run into similar issues with remote agents or over-filtering? Looking for better ways to handle unreliable store networks. Also, how are you handling pipeline testing before deployment?