You've correctly identified the parsed billing variable as the biggest risk. At your scale, with verbose auth logs, expect a 1.7x to 2x multiplier. That puts you near 1 TB/day billed, which can erase the per-GB savings.
For SPL-heavy queries, the issue isn't raw performance but compatibility. Complex joins and streamstats often require a full rewrite in their query language, which lacks equivalent functions. This creates a hidden migration cost.
The operational tax is managing the pre-filtering pipeline. The K8s collector is simple, but you'll need a dedicated Fluentd or OpenTelemetry layer to strip verbose fields before ingestion. That becomes a new platform to maintain.
That hidden migration cost for SPL rewrites is real. We underestimated the effort needed to rebuild our correlation searches using their available operators. It wasn't just a syntax change, it often required a completely different analytical approach.
The filtering layer you mentioned is key. It becomes a permanent, critical part of your stack. You're not just deploying collectors, you're building and owning a parsing gateway that dictates both your data quality and your monthly bill.
—Anita
That two-tier dashboard approach is exactly what the API limits force you into. The real kicker isn't the 5 minute lag, it's the architectural lock-in. Once you build for scheduled summaries, you can't just decide to run an ad-hoc investigation on a fresh data view without hitting throttles. Your SOC isn't just accepting a lag, they're giving up exploratory flexibility permanently.
Your CRM is lying to you.